RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control

[](https://huggingface.co/spaces/fffiloni/RB-Modulation) [](https://github.com/google/RB-Modulation)

Official PyTorch implementation of RB-Modulation: Training-Free Personalization of Diffusion Models using Stochastic Optimal Control.

Given reference images of preferred style or content, our method, RB-Modulation, offers a plug-and-play solution for (a) stylization with various prompts, and (b)
composition with reference content images while maintaining sample diversity and prompt alignment.

🔥 Updates

• [x] [2024.09.02] RB-Modulation Demo on Hugging Face! Thanks Sylvain Filoni.
• [x] [2024.08.23] RB-Modulation Code Release!
• [x] [2024.05.29] Paper is published on arXiv!

📥 Installation

# Download pretrained models.
cd third_party/StableCascade/models
bash download_models.sh essential big-big bfloat16
cd ..

# Install dependencies following the original [StableCascade](https://github.com/Stability-AI/StableCascade/blob/master/inference/readme.md)
conda create -n rbm python==3.9
pip install -r requirements.txt
pip install jupyter notebook opencv-python matplotlib ftfy

# Download [pre-trained CSD weights](https://drive.google.com/file/d/1FX0xs8p-C7Ob-h5Y4cUhTeOepHzXv_46/view) and put it under `third_party/CSD/checkpoint.pth`.

# Install LangSAM
pip install  git+https://github.com/IDEA-Research/GroundingDINO.git
pip install segment-anything==1.0
git clone https://github.com/luca-medeiros/lang-segment-anything && cd lang-segment-anything
pip install -e .

🚀 Try it!

jupyter notebook rb-modulation.ipynb

🤗 Gradio interface

We also support a Gradio <a href='https://github.com/gradio-app/gradio'>

</a> interface for better experience:
Web demonstration🔥

# Make sure you have the docker correctly setup.
git clone https://huggingface.co/spaces/fffiloni/RB-Modulation
cd RB-Modulation
python app.py

Citation

@inproceedings{
rout2025rbmodulation,
title={{RB}-Modulation: Training-Free Stylization using Reference-Based Modulation},
author={Litu Rout and Yujia Chen and Nataniel Ruiz and Abhishek Kumar and Constantine Caramanis and Sanjay Shakkottai and Wen-Sheng Chu},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=bnINPG5A32}
}

Disclaimer

This is not an officially supported Google product.