Benchmark and Evaluations, RL Alignment, Applications, and Challenges of Large Vision Language Models

A most Frontend Collection and survey of vision-language model papers, and models GitHub repository

Below we compile awesome papers and model and github repositories that

• State-of-the-Art VLMs Collection of newest to oldest VLMs (we'll keep updating new models and benchmarks).
• Evaluate VLM benchmarks and corresponding link to the works
• Post-training/Alignment Newest related work for VLM alignment including RL, sft.
• Applications applications of VLMs in embodied AI, robotics, etc.
• Contribute surveys, perspectives, and datasets on the above topics.

Welcome to contribute and discuss!

🤩 Papers marked with a ⭐️ are contributed by the maintainers of this repository. If you find them useful, we would greatly appreciate it if you could give the repository a star or cite our paper.

Table of Contents

• 📄 Paper Link/⛑️ Citation

• 1. 📚 SoTA VLMs
• 2. 🗂️ Dataset and Evaluation
  • 2.1. Large Scale Pre-Training & Post-Training Dataset
  • 2.2. Datasets and Evaluation for VLM
  • 2.3. Benchmark Datasets, Simulators and Generative Models for Embodied VLM

• 3. 🔥 Post-Training/Alignment/prompt engineering 🔥

  • 3.1. RL Alignment for VLM
  • 3.2. Regular finetuning (SFT)
  • 3.3. VLM Alignment Github
  • 3.4. Prompt Engineering
• 4. ⚒️ Applications
  • 4.1. Embodied VLM agents
  • 4.2. Generative Visual Media Applications
  • 4.3. Robotics and Embodied AI
    • 4.3.1. Manipulation
    • 4.3.2. Navigation
    • 4.3.3. Human-robot Interaction
    • 4.3.4. Autonomous Driving
  • 4.4. Human-Centered AI
    • 4.4.1. Web Agent
    • 4.4.2. Accessibility
    • 4.4.3. Medical and Healthcare
    • 4.4.4. Social Goodness
• 5. ⛑️ Challenges
  • 5.1. Hallucination
  • 5.2. Safety
  • 5.3. Fairness
  • 5.4. Alignment
    • 5.4.1. Multi-modality Alignment
      • 5.4.2. Commonsense and Physics Alignment
  • 5.5. Efficient Training and Fine-Tuning
  • 5.6. Scarce of High-quality Dataset

0. <a name='Citations'></a>Citation

@InProceedings{Li_2025_CVPR,
    author    = {Li, Zongxia and Wu, Xiyang and Du, Hongyang and Liu, Fuxiao and Nghiem, Huy and Shi, Guangyao},
    title     = {A Survey of State of the Art Large Vision Language Models: Benchmark Evaluations and Challenges},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2025},
    pages     = {1587-1606}
}

1. <a name='vlms'></a>📚 SoTA VLMs

2. <a name='Dataset'></a>🗂️ Benchmarks and Evaluation

2.1. <a name='TrainingDatasetforVLM'></a> Datasets for Training VLMs

2.2. <a name='DatasetforVLM'></a> Datasets and Evaluation for VLM

🧮 Visual Math (+ Visual Math Reasoning)

💬 Benchmark for Unified Models

🎞️ Video Understanding

💬 Multimodal Conversation

🧠 Multimodal General Intelligence

🔎 Visual Reasoning / VQA (+ Multilingual & OCR)

📝 Visual Text / Document Understanding (+ Charts)

🌄 Text‑to‑Image Generation

🚨 Hallucination Detection / Control

2.3. <a name='DatasetforEmbodiedVLM'></a> Benchmark Datasets, Simulators, and Generative Models for Embodied VLM

3. <a name='posttraining'></a>⚒️ Post-Training

3.1. <a name='alignment'></a>RL Alignment for VLM

3.2. <a name='sft'></a>Finetuning for VLM

3.3. <a name='vlm_github'></a>VLM Alignment github

3.4. <a name='vlm_prompt_engineering'></a>Prompt Optimization

4. <a name='Toolenhancement'></a> ⚒️ Applications

4.1 Embodied VLM Agents

4.2. <a name='GenerativeVisualMediaApplications'></a>Generative Visual Media Applications

4.3. <a name='RoboticsandEmbodiedAI'></a>Robotics and Embodied AI

4.3.1. <a name='Manipulation'></a>Manipulation

4.3.2. <a name='Navigation'></a>Navigation

4.3.3. <a name='HumanRobotInteraction'></a>Human-robot Interaction

4.3.4. <a name='AutonomousDriving'></a>Autonomous Driving

4.4. <a name='Human-CenteredAI'></a>Human-Centered AI

4.4.1. <a name='WebAgent'></a>Web Agent

4.4.2. <a name='Accessibility'></a>Accessibility

4.4.3. <a name='Medical and Healthcare'></a>Healthcare

4.4.4. <a name='SocialGoodness'></a>Social Goodness

5. <a name='Challenges'></a>Challenges

5.1 <a name='Hallucination'></a>Hallucination

5.2 <a name='Safety'></a>Safety

5.3 <a name='Fairness'></a>Fairness