Publication | Conference on Neural Information Processing Systems 2021
Program Synthesis Guided Reinforcement Learning
This Autodesk Research paper focuses on robotics in manufacturing. Users want to control these robots not by programming “how” to move but rather using high-level commands on “what” needs to be done. This paper outlines a way to automatically translate high level goal specifications to low level actuations
Download publicationAbstract
Program Synthesis Guided Reinforcement Learning
Yichen Yang, Jeevana Priya Inala, Osbert Bastani, Yewen Pu, Armando Solar-Lezama, Martin Rinard
Conference on Neural Information Processing Systems 2021
A key challenge for reinforcement learning is solving long-horizon planning and control problems. Recent work has proposed leveraging programs to help guide the learning algorithm in these settings. However, these approaches impose a high manual burden on the user since they must provide a guiding program for every new task they seek to achieve. We propose an approach that leverages program synthesis to automatically generate the guiding program. A key challenge is how to handle partially observable environments. We propose model predictive program synthesis, which trains a generative model to predict the unobserved portions of the world, and then synthesizes a program based on samples from this model in a way that is robust to its uncertainty. We evaluate our approach on a set of challenging benchmarks, including a 2D Minecraft-inspired “craft” environment where the agent must perform a complex sequence of subtasks to achieve its goal, a box-world environment that requires abstract reasoning, and a variant of the craft environment where the agent is a MuJoCo Ant. Our approach significantly outperforms several baselines, and performs essentially as well as an oracle that is given an effective program.
Associated Researchers
Yichen Yang
MIT
Jeevana Priya Inala
MIT
Armando Solar-Lezama
MIT
Martin Rinard
MIT
Osbert Bastani
University of Pennsylvania
Related Resources
2024
CalWave: Harnessing Ocean Power for a Sustainable FutureLearn how CalWave is advancing renewable energy sector with its wave…
2015
Tactum: A Skin-Centric Approach to Digital Design and FabricationSkin-based input has become an increasingly viable interaction model…
1993
The Design and Evaluation of Marking MenusThis research focuses on the use of hand drawn marks as a…
2022
It’s Over There: Designing an Intelligent Virtual Agent That Can Point Accurately into the Real WorldIt is challenging to design an intelligent virtual agent (IVA) that…
Get in touch
Something pique your interest? Get in touch if you’d like to learn more about Autodesk Research, our projects, people, and potential collaboration opportunities.
Contact us