Yilun Du

I am a fourth-year PhD student at MIT EECS, advised by Prof. Leslie Kaelbling, Prof. Tomas Lozano-Perez and Prof. Joshua B. Tenenbaum. Previously, I obtained my bachelor's degree from MIT, worked at OpenAI, Deepmind and FAIR, and got a gold medal at the International Biology Olympiad.

I am interested in constructing machine learning tools that enable the development of autonomous embodied agents. In the embodied setting, the world is richly combinatorical in nature and constantly evolving over time. To address these challenges, my recent research uses the tools of energy-based models to accurately generatively model the world and as a tool to construct composable models which may be rapidly adapted to new experiences. My research further uses the underlying energy optimization procedure as an adjustable computational budget, enabling the use of longer computation times to adapt to novel out-of-distribution experiences. Furthermore, embodied learning is richly multimodal in nature, and we need models which universally capture structure across modalities such as vision, text, sound and touch. I am interested in leveraging neural fields as a generic way to discover and capture such rich structure in the world. Finally I'm interested in broader applications of these tools to other domains such as computational biology.

News
Research Highlights
  • Generative Modeling: constructing generative models of the world's structure which may combined and adapted.
  • Perception and Scene Understanding: inferring structured representations of the world for downstream embodied tasks.
  • Interactive Learning: building intelligent agents which may interact in the surrounding world.
Publications ( show selected / show all by date / show all by topic )

Topics: Generative Modeling / Perception and Scene Understanding / Interactive Learning (* indicates equal contribution)

Is Conditional Generative Modeling all You Need for Decision-Making
Anurag Ajay*, Yilun Du*, Ahbi Gupta*, Joshua B. Tenenbaum, Tommi S. Jaakkola, Pulkit Agrawal

ArXiv Preprint / Website / Paper / Code

Composing Ensembles of Pre-trained Models via Iterative Consensus
Shuang Li*, Yilun Du*, Joshua B. Tenenbaum, Antonio Torralba, Igor Mordatch,

ArXiv Preprint / Website / Paper

Self-conditioned Embedding Diffusion for Text Generation
Robin Strudel, Corentin Tallec, Florent Altche, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, Remi Lebond

ArXiv Preprint / Paper

Seeing 3D Objects in a Single Image via Self-Supervised Static-Dynamic Disentanglement
Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Ambrus, Adrien Gaidon, William T. Freeman, Fredo Durand, Joshua B. Tenenbaum, Vincent Sitzmann

ArXiv Preprint / Website / Paper

SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields
Anthony Simeonov*, Yilun Du*, Yen-Chen Lin, Alberto Rodriguez, Leslie Kaelbling, Tomas Lozano-Perez, Antonio Torralba, Pulkit Agrawal

CoRL 2022 / Website / Paper / Code

MIRA: Mental Imagery for Robotic Affordances
Yen-Chen Lin, Pete Florence, Andy Zheng, Johnathon T. Barron, Yilun Du, Wei-Chiu Ma, Anthony Simeonov, Alberto Rodriguez, Phillip Isola

CoRL 2022 / Paper

Learning Neural Acoustic Fields
Andrew Luo, Yilun Du, Michael J. Tarr, Joshua B. Tenenbaum, Antonio Torralba, Chuang Gan

NeurIPS 2022 / Website / Paper / Code / Colab

3D Concept Grounding on Neural Fields
Yining Hong, Yilun Du, Chunru Lin, Joshua B. Tenenbaum, Chuang Gan

NeurIPS 2022 / Website / Paper

Pre-Trained Language Models for Interactive Decision-Making
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyurek, Anima Anandkumar+, Jacob Andreas+, Igor Mordatch+, Antonio Torralba+, Yuke Zhu+

NeurIPS 2022 (Oral) / Website / Paper / Code

(Last five authors contributed equally; order determined by alphabetically.)
Compositional Visual Generation with Composable Diffusion Models
Nan Liu*, Shuang Li*, Yilun Du*, Antonio Torralba, Joshua B. Tenenbaum

ECCV 2022 / Website / Paper / Code / Colab

Learning Object Based State Estimators for Household Autonomy
Yilun Du, Tomas Lozano-Perez, Leslie Kaelbling

IROS 2022 / Website / Paper

Robust Change Detection Based on Neural Descriptor Fields
Jiahui Fu, Yilun Du, Kurran Singh, Joshua B. Tenenbaum, John J. Leonard

IROS 2022 / Website / Paper

Planning with Diffusion for Flexible Behavior Synthesis
Michael Janner*, Yilun Du*, Joshua B. Tenenbaum, Sergey Levine

ICML 2022 (Long Talk) / Website / Paper / Code / Colab

Learning Iterative Reasoning through Energy Minimization
Yilun Du, Shuang Li, Joshua B. Tenenbaum, Igor Mordatch

ICML 2022 / Website / Paper / Code

Streaming Inference for Infinite Feature Models
Rylan Schaeffer, Yilun Du, Gabrielle Liu, Ila Fiete

ICML 2022 / Paper

Energy-Based Models for Continual Learning
Shuang Li, Yilun Du, Gido M. van de Ven, Antonio Torralba, Igor Mordatch

CoLLA 2022 (Oral) / Paper / Project Page / Code

Streaming Inference for Infinite Non-Stationary Clustering
Rylan Schaeffer, Gabrielle Liu, Yilun Du, Scott Linderman, Ila Fiete

CoLLA 2022 / Paper

Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation
Anthony Simeonov*, Yilun Du*, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal+, Vincent Sitzmann+

ICRA 2022 / Website / Paper / Code / Colab

(First two authors contributed equally; order determined by coin toss. Last two authors equal advising.)
Unsupervised Learning of Compositional Energy Concepts
Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, Igor Mordatch

NeurIPS 2021 / Website / Paper / Code

Learning to Compose Visual Relations
Nan Liu*, Shuang Li*, Yilun Du*, Joshua B. Tenenbaum, Antonio Torralba

NeurIPS 2021 (Spotlight) / NeurIPS 2021 Workshop on Controllable Generative Modeling (Outstanding Paper Award) / Website / Paper / Code / MIT News

Learning Signal-Agnostic Manifolds of Neural Fields
Yilun Du, Katie Collins, Joshua B. Tenenbaum, Vincent Sitzmann

NeurIPS 2021 / Website / Paper / Code

The Neural MMO Platform for Massively Multiagent Research
Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, Phillip Isola

NeurIPS 2021 Track on Datasets and Benchmarks / Website / Paper

3D Shape Generation and Completion through Point-Voxel Diffusion
Linqi Zhou, Yilun Du, Jiajun Wu

ICCV 2021 (Oral) / Project Page / Paper / Code

Curious Representation Learning for Embodied Intelligence
Yilun Du, Chuang Gan, Phillip Isola

ICCV 2021 / Project Page / Paper / Code

Neural Radiance Flow for 4D View Synthesis and Video Processing
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, Jiajun Wu

ICCV 2021 / Paper / Project Page / Code

Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Shuang Li, Yilun Du, Antonio Torralba, Josef Sivic, Bryan Russell

ICCV 2021 / Paper / Project Page / Code / Dataset

Improved Contrastive Divergence Training of Energy Based Models
Yilun Du, Shuang Li, Joshua B. Tenenbaum, Igor Mordatch

ICML 2021 / ICLR 2021 EBM Workshop (Oral) / Paper / Project Page / Code

Unsupervised Discovery of 3D Physical Objects from Video
Yilun Du, Kevin Smith, Tomer Ulman, Joshua B. Tenenbaum, Jiajun Wu

ICLR 2021 / Paper / Code / Project Page

Compositional Visual Generation with Energy Based Models
Yilun Du, Shuang Li, Igor Mordatch

NeurIPS 2020 (Spotlight) / Paper / Project Page / Code

A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois Hogan, Joshua B. Tenenbaum, Pulkit Agrawal, Alberto Rodriguez

CORL 2020 / Paper

Energy-based models for atomic-resolution protein conformations
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, Alexander Rives

ICLR 2020 (Spotlight) / MLCB 2020 (Oral / Travel Award) / Paper / Code

Observational Overfitting in Reinforcement Learning
Xingyou Song, Yiding Jiang, Yilun Du, Behnam Neyshabur

ICLR 2020 / Paper

Model Based Planning with Energy Based Models
Yilun Du, Toru Lin, Igor Mordatch

CORL 2019 / ICML MBRL Workshop 2019 (Oral) / Paper / Code

Implicit Generation and Generalization with Energy Based Models
Yilun Du, Igor Mordatch

NeurIPS 2019 (Spotlight) / Paper / Code / OpenAI Blog

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
Yilun Du, Karthik Narasimhan

ICML 2019 / Paper / Code

Neural MMO: A massively multiplayer game environment for intelligent agents
Joseph Suarez, Yilun Du, Phillip Isola, Igor Mordatch

AAMAS 2020 Extended Abstract / Paper / Code / OpenAI Blog

Learning to Exploit Stability for 3D Scene Parsing
Yilun Du, Zhijian Liu, Hector Basevi, Ales Leonardis, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu

NeurIPS 2018 / Paper