✨✨VLADBench✨✨: Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving

📖 arXiv Paper      🤗 Dataset 🔧 Support VLMEvalKit

Overview

Current benchmarks designed for VLM-based AD face several notable limitations:

• Coarse-grained Categories: The underlying datasets of the VLM-based models are often simplistic, typically categorizing tasks into perception, prediction, and planning with reasoning, which are incomplete for evaluating the nuanced cognitive and reasoning abilities required for safe and reliable AD.

• Lack of Dynamic Elements: Both static and dynamic scenes are crucial for evaluating AD systems, a robust analysis of dynamic elements is particularly important for validating the temporal reasoning capabilities, especially in understanding traffic participant intentions within the scene and executing the nuanced spatio-temporal reasoning required for safe navigation.

• Homogeneous Data: Existing VLM-based AD datasets often suffer from a lack of diversity, which limits the ability to test models across a wide range of real-world scenarios. The narrow results restrict the evaluation of zero-shot generalization and the performance on challenging corner cases.

We introduce VLADBench, specifically designed to rigorously evaluate the capabilities of VLMs in AD. VLADBench employes a hierarchical structure that reflects the complex skill set required for reliable driving, progressing from fundamental scene and traffic elements comprehension to advanced reasoning and decision-making.

• With 2000 static scenes and 3000 dynamic scenarios, VLADBench spans 5 primary domains: Traffic Knowledge Understanding (TKU), General Element Recognition (GER), Traffic Graph Generation (TGG), Target Attribute Comprehension (TAC), and Ego Decision-making and Planning(EDP). For a more detailed assessment, 11 secondary aspects and 29 tertiary tasks are defined, resulting in a total of 12K questions.
• VLADBench is built from existing publicly available datasets, meticulously curated through a manual selection across 12 sources, aimed at challenging VLM capabilities in diverse challenging driving situations.
• To further investigate the intersections among the 5 key domains, we collect and construct approximately 1.4M AD-specific QAs from public resources. We then categorize these QAs using GPT-4 and train models on individual domain-specific (DS) datasets. Finally, we validate the trained models on VLADBench to assess their performance across different domains.

Examples

Evaluation Results

For the detailed results of each tasks and the results from the large-scale models, please see the paper.

Evaluation Pipeine

1. We provide a test example based on Qwen2-VL.[qwen2vl_all.py]
2. Run evaluate_vlm.py for scores.

Note
1. The bounding boxes in VLADBench are NOT resized. You should modify them in prompt and evaluation for different VLMs.
2. The finall scores do NOT include the trajectory evaluation.

Running the benchmark

pip install -e .
vladbench run results/protocols/full-original.json --models inkling      # billable
vladbench score results/protocols/full-original.json --models inkling

results/protocols/full-original.json is the whole protocol: dataset revision, prompt handling, completion guard, reasoning policy, and one entry per model declaring its endpoint, sequence transport, and reasoning setting. The runner sends each unanswered question and appends the answer under results/runs/; the scorer rebuilds every request, checks the stored hash, and applies the paper's scoring criteria, running its scoring code unchanged. Details and the claims the data supports are in docs/PROTOCOL.md; commands in docs/REPRODUCTION.md.

Layout:

• src/vladbench/: dataset, spec, requests, video, run, scoring, cli.
• results/: everything about a condition in one place. protocols/ holds the specification (one file is one condition), runs/ the raw per-question answers (gitignored; answers/<Task>.json at the repo root carries every model's answer and its per-question mark for the results page, built by scripts/build_answers.py), scores-<model>.json the scored summaries, audit/ the pinned dataset metadata and the published Table 10 values, and rerun.json the compact record the results page reads.
• original/: the paper's scoring code (its criteria, unchanged), task catalog, and README images, byte for byte, with hashes.
• results.html (tabbed results: cost against score with the frontier and a video-cost calculator, the paper's Table 10 layout, score by task, task matrix), task-review.html (the older single-page companion) and self-test.html (take the benchmark yourself): static pages, python3 -m http.server. scripts/export_parquet.py writes the published parquet tables to results/dataset/; scripts/publish_hf.py pushes them to the Hugging Face dataset (--with-space also publishes the static site), sending only changed files; it needs HF_TOKEN in .env. scripts/build_blog_embed.py assembles the bundle the blog serves.
• scripts/: the results builder, the parquet export, the publishing and blog-bundle scripts, the metering re-fit report, and the dataset audit.
• src/vladbench/metering.py: the fitted tokeniser and billing rules behind the cost-per-video-hour estimates, validated offline against every sweep and live against the API in tests/test_metering.py (set RUN_LIVE_METERING=1).