Multimodality in Robotics Explained

This repository contains the companion Jupyter notebook for the video "Multimodality in Robotics" that explores different approaches to handling multimodality in robotics: predicting continuous actions when multiple correct actions exist.

Overview

The notebook demonstrates several approaches to handling multimodal distributions in robotics:

1. Simple MSE regression and its limitations
2. MAE regression and comparison with MSE
3. Target tokenization and classification
4. Diffusion Policy
5. Flow Matching Policy

Each approach is implemented with clear code examples and visualizations to help understand the concepts.

Getting Started

Running the Notebook locally

1. Clone this repository:

   git clone https://github.com/IliaLarchenko/robotics_multimodality_explained
   cd robotics_multimodality_explained

2. Install the required packages:

   pip install -r requirements.txt

3. Open and run the notebook:

   jupyter notebook multimodality_analysis.ipynb

Running on Kaggle

The notebook is also available on Kaggle, just click the link below:
Kaggle Notebook Version

References

Papers

• ACT: Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
• Behavior Transformers: Cloning k Modes with One Stone
• VQ-BET: Behavior Generation with Latent Actions
• Denoising Diffusion Probabilistic Models
• DDIM: Denoising Diffusion Implicit Models
• Flow Matching for Generative Modeling

Additional Resources

• MIT Course on Diffusion & Flow Matching (SDE perspective)
• DOT-Policy Implementation

Author

Ilia Larchenko

• GitHub: @IliaLarchenko
• Kaggle: @ilialar
• YouTube: @IliaLarchenko
• Linktree: @IliaLarchenko