Yilun Du

I am a third-year PhD student at MIT EECS, advised by Prof. Leslie Kaelbling, Prof. Tomas Lozano-Perez and Prof. Josh Tenenbaum. I am interested in building robotics agents which may perceive and understand the world as humans do, and are able to construct world representations that enable long horizon robotic task planning and manipulation in partially observable environments. I am also interested in constructing modular systems which may be incrementally learned and composed to enable robust test time generalization. My research is driven by the goal of constructing intelligent autonomous household robotics - robots which can incrementally learn and accomplish common tasks, such as finding an object or cooking some food, tasks that are trivial for humans, but remain impossible for current AI systems. Previously, I obtained my bachelor's degree from MIT, worked at OpenAI and FAIR, and got a gold medal at the International Biology Olympiad.

Research Highlights
  • Perception and Scene Understanding: learning flexible representations of the world (both at the individual object level and scene level) that enable effective downstream mobile robotic manipulation.
  • Generative Modeling: modeling the underlying state of the world in a composable, modular and flexible manner, as well as different applications of resultant models.
  • Interactive Learning: learning to build intelligent agents which may interact in the world around them.
Publications ( show selected / show all by date / show all by topic )

Topics: Generative Modeling / Perception and Scene Understanding / Interactive Learning (* indicates equal contribution)

Unsupervised Learning of Compositional Energy Concepts
Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, Igor Mordatch

NeurIPS 2021 / Website / Paper / Code

Learning Signal-Agnostic Manifolds of Neural Fields
Yilun Du, Katie Collins, Joshua B. Tenenbaum, Vincent Sitzmann

NeurIPS 2021 / Website / Paper / Code

Learning to Compose Visual Relations
Nan Liu*, Shuang Li*, Yilun Du*, Joshua B. Tenenbaum, Antonio Torralba

NeurIPS 2021 (Spotlight) / Website / Paper / Code

The Neural MMO Platform for Massively Multiagent Research
Joseph Suarez, Yilun Du, Clare Zhu, Igor Mordatch, Phillip Isola

NeurIPS 2021 Track on Datasets and Benchmarks / Website / Paper

3D Shape Generation and Completion through Point-Voxel Diffusion
Linqi Zhou, Yilun Du, Jiajun Wu

ICCV 2021 (Oral) / Project Page / Paper / Code

Curious Representation Learning for Embodied Intelligence
Yilun Du, Chuang Gan, Phillip Isola

ICCV 2021 / Project Page / Paper / Code

Neural Radiance Flow for 4D View Synthesis and Video Processing
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, Jiajun Wu

ICCV 2021 / Paper / Project Page / Code

Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Shuang Li, Yilun Du, Antonio Torralba, Josef Sivic, Bryan Russell

ICCV 2021 / Paper / Project Page / Code / Dataset

Improved Contrastive Divergence Training of Energy Based Models
Yilun Du, Shuang Li, Joshua B. Tenenbaum, Igor Mordatch

ICML 2021 / ICLR 2021 EBM Workshop (Oral) / Paper / Project Page / Code

Energy-Based Models for Continual Learning
Shuang Li, Yilun Du, Gido M. van de Ven, Antonio Torralba, Igor Mordatch

ICLR 2021 EBM Workshop (Oral) / Paper / Project Page

Unsupervised Discovery of 3D Physical Objects from Video
Yilun Du, Kevin Smith, Tomer Ulman, Joshua B. Tenenbaum, Jiajun Wu

ICLR 2021 / Paper / Project Page

Learning an Object Based Memory System
Yilun Du, Joshua B. Tenenbaum, Tomas Lozano-Perez, Leslie Kaelbling

ArXiv Preprint / Paper

Compositional Visual Generation with Energy Based Models
Yilun Du, Shuang Li, Igor Mordatch

NeurIPS 2020 (Spotlight) / Paper / Project Page / Code

A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois Hogan, Joshua B. Tenenbaum, Pulkit Agrawal, Alberto Rodriguez

CORL 2020 / Paper

Energy-based models for atomic-resolution protein conformations
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, Alexander Rives

ICLR 2020 (Spotlight) / MLCB 2020 (Oral / Travel Award) / Paper / Code

Observational Overfitting in Reinforcement Learning
Xingyou Song, Yiding Jiang, Yilun Du, Behnam Neyshabur

ICLR 2020 / Paper

Model Based Planning with Energy Based Models
Yilun Du, Toru Lin, Igor Mordatch

CORL 2019 / ICML MBRL Workshop 2019 (Oral) / Paper / Code

Implicit Generation and Generalization with Energy Based Models
Yilun Du, Igor Mordatch

NeurIPS 2019 (Spotlight) / Paper / Code / OpenAI Blog

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
Yilun Du, Karthik Narasimhan

ICML 2019 / Paper / Code

Neural MMO: A massively multiplayer game environment for intelligent agents
Joseph Suarez, Yilun Du, Phillip Isola, Igor Mordatch

AAMAS 2020 Extended Abstract / Paper / Code / OpenAI Blog

Learning to Exploit Stability for 3D Scene Parsing
Yilun Du, Zhijian Liu, Hector Basevi, Ales Leonardis, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu

NeurIPS 2018 / Paper