Unsupervised Image to Sequence Translation with Canvas-Drawer Networks


Unsupervised Image to Sequence Translation with Canvas-Drawer Networks

Kevin Frans, Chin-Yi Cheng

Encoding images as a series of high-level constructs, such as brush strokes or discrete shapes, can often be key to both human and machine understanding. In many cases, however, data is only available in pixel form. We present a method for generating images directly in a high-level domain (e.g. brush strokes), without the need for real pairwise data. Specifically, we train a ”canvas” network to imitate the mapping of high-level constructs to pixels, followed by a high-level ”drawing” network which is optimized through this mapping towards solving a desired image recreation or translation task. We successfully discover sequential vector representations of symbols, large sketches, and 3D objects, utilizing only pixel data. We display applications of our method in image segmentation, and present several ablation studies comparing various configurations.

Download publication

Associated Researchers

Chin-Yi Cheng

Autodesk Research

Kevin Frans

Massachusetts Institute of Technology

View all researchers

Related Resources



Leveraging Graph Neural Networks for Graph Regression and Effective Enumeration Reduction

Graph-based framework represents aspects of optimal thermal management…



The Limits of Expert Performance Using Hierarchic Marking Menus

A marking menu allows a user to perform a menu selection by either…



Reducing Cognitive Bias in Biomimetic Design by Abstracting Nouns

Biological analogies can increase creativity in design by providing…



On Subdivision Schemes Generalizing Uniform B-spline Surfaces of Arbitrary Degree

We introduce a new class of subdivision surfaces which generalize…

Get in touch

Something pique your interest? Get in touch if you’d like to learn more about Autodesk Research, our projects, people, and potential collaboration opportunities.

Contact us