About Yilun Du

We study how intelligent systems can learn reusable models, then compose and reason with them at inference time.

Energy-based and diffusion models are our main tools for capturing and composing this structure. We have studied these ideas across generation, reasoning, and decision making, including systems that coordinate multiple models or agents. A central goal of our research is to build robots that can combine knowledge of objects, dynamics, and skills to plan and act in unfamiliar environments.

I lead the Embodied Minds Lab at Harvard and am a part-time research scientist at NVIDIA Research. I received my PhD from MIT EECS, advised by Leslie Kaelbling, Tomas Lozano-Pérez, and Joshua B. Tenenbaum. Before Harvard, I held research roles at OpenAI and Google DeepMind.

Research

Generative Inference

KnowledgeSolution spacesReasoningSample and refine

We study how to construct and refine solutions using a learned generative model. Sampling and optimization use the model to propose candidates and improve them. We develop representations and inference procedures to make this search reliable and adapt computation to the problem.

Examples: Reasoning with Sampling improves language-model reasoning by changing the sampling procedure while keeping the model fixed. Deep EBMs learn energy landscapes for generation and conditional inference. Reduce, Reuse, Recycle develops samplers for composing pretrained diffusion models.

Compositional Inference

KnowledgeConcepts, constraints, and skillsReasoningCompose and solve

We study how to combine learned models and task requirements to define new inference problems. Models of concepts, constraints, and skills provide reusable components, while inference searches for solutions that satisfy the combined requirements. This lets us tackle new combinations without learning a separate model for each complete task.

Examples: Composable diffusion models combine learned concepts to generate unfamiliar combinations. Diffuser reuses a trajectory model under different goals and constraints. Compositional energy minimization assembles subproblem energies to solve larger reasoning problems.

World Models & Robotics

KnowledgeWorld dynamicsReasoningPredict and plan

We study how robots can use learned world models to plan and act in unfamiliar environments. These models predict the consequences of possible actions, while planning searches for actions that achieve the current goal. This lets robots reuse the same knowledge across tasks and adjust their behavior as goals and environments change.

Examples: UniPi generates video plans that can be translated into actions. Large Video Planner uses a pretrained video model to guide robot control. RAX lets a coding agent improve robot control through online exploration, saving what works as reusable programs.

Multi-Agent Intelligence

KnowledgeDistributed across agentsReasoningDebate and verify

We study multiagent computation as a way to scale language-model intelligence. Rather than relying only on stronger individual models or longer reasoning traces, we train complementary agents and organize how they explore, verify, and combine solutions. The central question is how additional collective computation can produce capabilities beyond those of individual models.

Examples: Multi-agent debate lets models critique and revise candidate answers. Multi-agent verification combines multiple verifiers to evaluate candidate solutions. Economy of Minds organizes agents through task-dependent economic interactions.

I talk about our research connecting these directions in Generalization by Construction, a research perspective on what learning should leave open for inference. My PhD thesis and thesis defense provide a description of our view through energy-based modeling, compositional inference, and planning. For an introduction to Energy-Based Models, see An Energy-Based Perspective on Machine Learning, a tutorial with Bo Dai.

News & updates

Publications

Topics:
Showing 35 of 35 selected publications

* Equal contribution; † equal advising.