Yilun Du

I am a second-year PhD student at MIT EECS, advised by Prof. Leslie Kaelbling and Prof. Tomas Lozano-Perez and Prof. Josh Tenenbaum. I am interested in learning state representations of the world that enable generalizable long horizon robotic task planning and manipulation given only perceptual observations, in settings where the world is both large and partially observable. My research is driven by the goal of constructing intelligent autonomous household robotics - robots which can accomplish common tasks, such as finding an object or cooking some food, tasks that are trivial for humans, but remain impossible for current AI systems. Previously, I obtained my bachelor's degree from MIT, worked at OpenAI and FAIR, and got a gold medal at the International Biology Olympiad.

Research Highlights
  • State Estimation: long term estimation of the state of mobile objects
  • Perception Grounded Manipulation: visually guided multi-step manipulation of object
  • Physical and 3D Scene Understanding: unsupervised object discovery, 3D reconstruction
  • Generative Modeling: energy-based models, compositionality, applications of generative models
  • Reinforcement Learning: multi-agent/model based/unsupervised reinforcement learning
Publications ( show selected / show all by date / show all by topic )

Topics: Generative Modeling / Physical and 3D Scene Understanding / State Estimation / Perception Grounded Manipulation / Reinforcement Learning (* indicates equal contribution)

3D Shape Generation and Completion through Point-Voxel Diffusion
Linqi Zhou, Yilun Du, Jiajun Wu

ICCV 2021 (Oral) / Project Page / Paper

Curious Representation Learning for Embodied Intelligence
Yilun Du, Chuang Gan, Phillip Isola

ICCV 2021 / Project Page / Paper

Neural Radiance Flow for 4D View Synthesis and Video Processing
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, Jiajun Wu

ICCV 2021 / Paper / Project Page

Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Shuang Li, Yilun Du, Antonio Torralba, Josef Sivic, Bryan Russell

ICCV 2021

Improved Contrastive Divergence Training of Energy Based Models
Yilun Du, Shuang Li, Joshua B. Tenenbaum, Igor Mordatch

ICML 2021 / ICLR 2021 EBM Workshop (Oral) / Paper / Project Page

Energy-Based Models for Continual Learning
Shuang Li, Yilun Du, Gido M. van de Ven, Antonio Torralba, Igor Mordatch

ICLR 2021 EBM Workshop (Oral) / Paper / Project Page

Unsupervised Discovery of 3D Physical Objects from Video
Yilun Du, Kevin Smith, Tomer Ulman, Joshua B. Tenenbaum, Jiajun Wu

ICLR 2021 / Paper / Project Page

Learning an Object Based Memory System
Yilun Du, Joshua B. Tenenbaum, Tomas Lozano-Perez, Leslie Kaelbling

ArXiv Preprint / Paper

Compositional Visual Generation with Energy Based Models
Yilun Du, Shuang Li, Igor Mordatch

NeurIPS 2020 (Spotlight) / Paper / Project Page / Code

A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois Hogan, Joshua B. Tenenbaum, Pulkit Agrawal, Alberto Rodriguez

CORL 2020 / Paper

Energy-based models for atomic-resolution protein conformations
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, Alexander Rives

ICLR 2020 (Spotlight) / MLCB 2020 (Oral / Travel Award) / Paper / Code

Observational Overfitting in Reinforcement Learning
Xingyou Song, Yiding Jiang, Yilun Du, Behnam Neyshabur

ICLR 2020 / Paper

Model Based Planning with Energy Based Models
Yilun Du, Toru Lin, Igor Mordatch

CORL 2019 / ICML MBRL Workshop 2019 (Oral) / Paper / Code

Implicit Generation and Generalization with Energy Based Models
Yilun Du, Igor Mordatch

NeurIPS 2019 (Spotlight) / Paper / Code / OpenAI Blog

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning
Yilun Du, Karthik Narasimhan

ICML 2019 / Paper / Code

Neural MMO: A massively multiplayer game environment for intelligent agents
Joseph Suarez, Yilun Du, Phillip Isola, Igor Mordatch

AAMAS 2020 Extended Abstract / Paper / Code / OpenAI Blog

Learning to Exploit Stability for 3D Scene Parsing
Yilun Du, Zhijian Liu, Hector Basevi, Ales Leonardis, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu

NeurIPS 2018 / Paper