Freehand Sketch Generation from Mechanical Components

★ Co-first authors (equal contribution), ✉ Correspending authors
1Institute of Data and Information, Tsinghua Shenzhen International Graduate School, Tsinghua University 2State Key Laboratory of Fluid Power & Mechatronic Systems, Zhejiang University

ACM MM 2024

Contributions of Main Authors

Zhichao Liao: 95% of paper and supporting material writing; 20% of literature review; paper submission follow-up.

Di Huang: 95% of ideas (task & core method design) and coding; 80% of literature review; paper writing guidance.

Heming Fang: Joint project support; view selector; partial experimental evaluation metric acquisition & prelim. study.

Fengyuan Piao: Data collection; discussion scheduling and progress follow-up; project alignment (meeting with partners).

Xinghui Li: Paper writing and polishing guidance, paper submission guidance.

Long Zeng: Paper polishing guidance, project support, computing resource support, funding.

Freehand Sketch Generation


Abstract

Drawing freehand sketches of mechanical components on multimedia devices for AI-based engineering modeling becomes a new trend. However, its development is being impeded because existing works cannot produce suitable sketches for data-driven research. These works either generate sketches lacking a freehand style or utilize generative models not originally designed for this task resulting in poor effectiveness. To address this issue, we design a two-stage generative framework mimicking the human sketching behavior pattern, called MSFormer, which is the first time to produce humanoid freehand sketches tailored for mechanical components. The first stage employs Open CASCADE technology to obtain multi-view contour sketches from mechanical components, filtering perturbing signals for the ensuing generation process. Meanwhile, we design a view selector to simulate viewpoint selection tasks during human sketching for picking out information-rich sketches. The second stage translates contour sketches into freehand sketches by a transformer-based generator. To retain essential modeling features as much as possible and rationalize stroke distribution, we introduce a novel edge-constraint stroke initialization. Furthermore, we utilize a CLIP vision encoder and a new loss function incorporating the Hausdorff distance to enhance the generalizability and robustness of the model. Extensive experiments demonstrate that our approach achieves state-of-the-art performance for generating freehand sketches in the mechanical domain.

Framework




BibTeX


      @misc{liao2024freehand,
        title={Freehand Sketch Generation from Mechanical Components},
        author={Liao, Zhichao and Huang, Di and Fang, Heming and Ma, Yue and Piao, Fengyuan and Li, Xinghui and Zeng, Long and Feng, Pingfa},
        year={2024},
        eprint={2408.05966},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2408.05966},
      }