Yilun Du

I am a fourth-year PhD student at MIT EECS, advised by Prof. Leslie Kaelbling, Prof. Tomas Lozano-Perez and Prof. Joshua B. Tenenbaum. Previously, I obtained my bachelor's degree from MIT, was a research fellow at OpenAI, an intern at Deepmind and FAIR, and got a gold medal at the International Biology Olympiad.

My research interests lie in developing machine learning tools to create autonomous embodied agents. Two major challenges presented in the embodied setting include: high uncertainty and the rich combinatorial nature of the world. To tackle these issues, I utilize energy-based models to model underlying probability distributions in this setting and to construct composable models that can flexibly adapted to new experience. I am also interested in using the iterative energy optimization procedure as an adjustable computational budget and as a means to combine existing large pretrained models. Embodied learning is multimodal, and I use neural fields as a tool to capture structure across modalities such as vision, text, sound, and touch. Finally I'm interested in broader applications of these tools to other domains such as computational biology.

News
  • I gave a recent talk at Nuro summarizing some of my recent works recorded here.
  • We are organizing a workshop on large pretrained models for decision making at NeurIPS 2022.
  • Check out a list of our work on energy-based models!
  • If you are looking for research experience, feel free to reach out -- we have several projects actively looking for additional collaborators.
Research Highlights
  • Generative Modeling: constructing generative models of the world's structure which may be combined and adapted.
  • Perception and Scene Understanding: inferring structured representations of the world for downstream embodied tasks.
  • Interactive Learning: building intelligent agents which may interact in the surrounding world.
Publications ( show selected / show all by date / show all by topic )

Topics: Generative Modeling / Perception and Scene Understanding / Interactive Learning (* indicates equal contribution and indicates equal advising)

Improving Factuality and Reasoning in Language Models through Multiagent Debate
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, Igor Mordatch

ArXiv Preprint / Website / Paper / Code

Training Diffusion Models with Reinforcement Learning
Kevin Black*, Michael Janner*, Yilun Du, Ilya Kostrikov, Sergey Levine

ArXiv Preprint / Website / Paper

Foundation Models for Decision Making: Problems, Methods, and Opportunities
Mengjiao Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, Dale Schuurmans

ArXiv Preprint / Paper

Learning Universal Policies via Text-Guided Video Generation
Yilun Du*, Mengjiao Yang*, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, Pieter Abbeel

ArXiv Preprint / Website / Paper

Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC
Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, Will Grathwohl

ICML 2023 / Website / Colab / Code / Paper

NeuSE: Neural SE(3)-Equivariant Embedding for Consistent Spatial Understanding with Objects
Jiahui Fu, Yilun Du, Kurran Singh, Joshua B. Tenenbaum, John J. Leonard

RSS 2023 / Website / Paper

StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects
Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, Chris Paxton

RSS 2023 / Website / Paper

Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Shuran Song

RSS 2023 / Website / Code / Paper

Learning to Render Novel Views from Wide-Baseline Stereo Pairs
Yilun Du, Cameron Smith, Ayush Tewari, Vincent Sitzmann

CVPR 2023 / Website / Code / Colab / Paper

3D Concept Learning and Reasoning from Multi-View Images
Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B. Tenenbaum, Chuang Gan

CVPR 2023 / Website / Paper

Is Conditional Generative Modeling all You Need for Decision-Making?
Anurag Ajay*, Yilun Du*, Ahbi Gupta*, Joshua B. Tenenbaum, Tommi S. Jaakkola, Pulkit Agrawal

ICLR 2023 (Top 5% Notable) / Website / Paper / Code

Composing Ensembles of Pre-trained Models via Iterative Consensus
Shuang Li*, Yilun Du*, Joshua B. Tenenbaum, Antonio Torralba, Igor Mordatch

ICLR 2023 / Website / Paper

Planning with Sequence Models through Iterative Energy Minimization
Hongyi Chen*, Yilun Du*, Yiye Chen*, Joshua B. Tenenbaum, Patricio Antonio Vela

ICLR 2023 / Website / Paper / Code

Seeing 3D Objects in a Single Image via Self-Supervised Static-Dynamic Disentanglement
Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Ambrus, Adrien Gaidon, William T. Freeman, Fredo Durand, Joshua B. Tenenbaum, Vincent Sitzmann

ICLR 2023 / Website / Paper

Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation
Ethan Chun, Yilun Du, Anthony Simeonov, Tomas Lozano-Perez, Leslie Kaelbling

ICRA 2023 / Paper / Website / Code

Visiblity-Aware Navigation Among Movable Objects
Jose Iturralde*, Aiden Curtis*, Yilun Du, Leslie Kaelbling, Tomas Lozano-Perez

ICRA 2023 / Paper

Language Models Generalize Beyond Natural Proteins
Robert Verkuil*, Ori Kabeli*, Yilun Du, Basile Wicky, Lukas Milles, Justas Dauparas, David Baker, Sergey Ovchinnikov, Tom Sercu, Alexander Rives

bioRxiv Preprint / Paper

Self-conditioned Embedding Diffusion for Text Generation
Robin Strudel, Corentin Tallec, Florent Altche, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, Remi Lebond

ArXiv Preprint / Paper

SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields
Anthony Simeonov*, Yilun Du*, Yen-Chen Lin, Alberto Rodriguez, Leslie Kaelbling, Tomas Lozano-Perez, Pulkit Agrawal

CoRL 2022 / Website / Paper / Code

MIRA: Mental Imagery for Robotic Affordances
Yen-Chen Lin, Pete Florence, Andy Zheng, Johnathon T. Barron, Yilun Du, Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez, Phillip Isola

CoRL 2022 / Paper/ Website

Learning Neural Acoustic Fields
Andrew Luo, Yilun Du, Michael J. Tarr, Joshua B. Tenenbaum, Antonio Torralba, Chuang Gan

NeurIPS 2022 / Website / Paper / Code / Colab

3D Concept Grounding on Neural Fields
Yining Hong, Yilun Du, Chunru Lin, Joshua B. Tenenbaum, Chuang Gan

NeurIPS 2022 / Website / Paper

Pre-Trained Language Models for Interactive Decision-Making
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyurek, Anima Anandkumar+, Jacob Andreas+, Igor Mordatch+, Antonio Torralba+, Yuke Zhu+

NeurIPS 2022 (Oral) / Website / Paper / Code

(Last five authors contributed equally; order determined by alphabetically.)
Compositional Visual Generation with Composable Diffusion Models
Nan Liu*, Shuang Li*, Yilun Du*, Antonio Torralba, Joshua B. Tenenbaum

ECCV 2022 / Website / Paper / Code / Colab

Learning Object Based State Estimators for Household Autonomy
Yilun Du, Tomas Lozano-Perez, Leslie Kaelbling

IROS 2022 / Website / Paper

Robust Change Detection Based on Neural Descriptor Fields
Jiahui Fu, Yilun Du, Kurran Singh, Joshua B. Tenenbaum, John J. Leonard

IROS 2022 / Website / Paper

Planning with Diffusion for Flexible Behavior Synthesis
Michael Janner*, Yilun Du*, Joshua B. Tenenbaum, Sergey Levine

ICML 2022 (Long Talk) / Website / Paper / Code / Colab

Learning Iterative Reasoning through Energy Minimization
Yilun Du, Shuang Li, Joshua B. Tenenbaum, Igor Mordatch

ICML 2022 / Website / Paper / Code

Streaming Inference for Infinite Feature Models
Rylan Schaeffer, Yilun Du, Gabrielle Liu, Ila Fiete

ICML 2022 / Paper

Energy-Based Models for Continual Learning
Shuang Li, Yilun Du, Gido M. van de Ven, Antonio Torralba, Igor Mordatch

CoLLA 2022 (Oral) / Paper / Project Page / Code

Streaming Inference for Infinite Non-Stationary Clustering
Rylan Schaeffer, Gabrielle Liu, Yilun Du, Scott Linderman, Ila Fiete

CoLLA 2022 / Paper

Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation
Anthony Simeonov*, Yilun Du*, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal+, Vincent Sitzmann+

ICRA 2022 / Website / Paper / Code / Colab

(First two authors contributed equally; order determined by coin toss. Last two authors equal advising.)
Unsupervised Learning of Compositional Energy Concepts
Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, Igor Mordatch

NeurIPS 2021 / Website / Paper / Code

Learning to Compose Visual Relations
Nan Liu*, Shuang Li*, Yilun Du*, Joshua B. Tenenbaum, Antonio Torralba

NeurIPS 2021 (Spotlight) / NeurIPS 2021 Workshop on Controllable Generative Modeling (Outstanding Paper Award) / Website / Paper / Code / MIT News

Learning Signal-Agnostic Manifolds of Neural Fields
Yilun Du, Katie Collins, Joshua B. Tenenbaum, Vincent Sitzmann

NeurIPS 2021 / Website / Paper / Code

The Neural MMO Platform for Massively Multiagent Research
Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, Phillip Isola

NeurIPS 2021 Track on Datasets and Benchmarks / Website / Paper

3D Shape Generation and Completion through Point-Voxel Diffusion
Linqi Zhou, Yilun Du, Jiajun Wu

ICCV 2021 (Oral) / Project Page / Paper / Code

Curious Representation Learning for Embodied Intelligence
Yilun Du, Chuang Gan, Phillip Isola

ICCV 2021 / Project Page / Paper / Code

Neural Radiance Flow for 4D View Synthesis and Video Processing
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, Jiajun Wu

ICCV 2021 / Paper / Project Page / Code

Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Shuang Li, Yilun Du, Antonio Torralba, Josef Sivic, Bryan Russell

ICCV 2021 / Paper / Project Page / Code / Dataset

Improved Contrastive Divergence Training of Energy Based Models
Yilun Du, Shuang Li, Joshua B. Tenenbaum, Igor Mordatch

ICML 2021 / ICLR 2021 EBM Workshop (Oral) / Paper / Project Page / Code

Unsupervised Discovery of 3D Physical Objects from Video
Yilun Du, Kevin Smith, Tomer Ulman, Joshua B. Tenenbaum, Jiajun Wu

ICLR 2021 / Paper / Code / Project Page

Compositional Visual Generation with Energy Based Models
Yilun Du, Shuang Li, Igor Mordatch

NeurIPS 2020 (Spotlight) / Paper / Project Page / Code

A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois Hogan, Joshua B. Tenenbaum, Pulkit Agrawal, Alberto Rodriguez

CORL 2020 / Paper

Energy-based models for atomic-resolution protein conformations
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, Alexander Rives

ICLR 2020 (Spotlight) / MLCB 2020 (Oral / Travel Award) / Paper / Code

Observational Overfitting in Reinforcement Learning
Xingyou Song, Yiding Jiang, Yilun Du, Behnam Neyshabur

ICLR 2020 / Paper

Model Based Planning with Energy Based Models
Yilun Du, Toru Lin, Igor Mordatch

CORL 2019 / ICML MBRL Workshop 2019 (Oral) / Paper / Code

Implicit Generation and Generalization with Energy Based Models
Yilun Du, Igor Mordatch

NeurIPS 2019 (Spotlight) / Paper / Code / OpenAI Blog

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
Yilun Du, Karthik Narasimhan

ICML 2019 / Paper / Code

Neural MMO: A massively multiplayer game environment for intelligent agents
Joseph Suarez, Yilun Du, Phillip Isola, Igor Mordatch

AAMAS 2020 Extended Abstract / Paper / Code / OpenAI Blog

Learning to Exploit Stability for 3D Scene Parsing
Yilun Du, Zhijian Liu, Hector Basevi, Ales Leonardis, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu

NeurIPS 2018 / Paper