VDiff-Bench-MCQ
Controlled identification
1
2Which difference is real?
Select one verified difference from two plausible alternatives and a no-difference distractor.
Two complementary tracks
VDiff-Bench separates controlled identification from open-ended discovery, exposing whether a model can recognize a difference and whether it can find every difference independently.
VDiff-Bench-MCQ
Controlled identification
1
2Select one verified difference from two plausible alternatives and a no-difference distractor.
VDiff-Bench-Gen
Open-ended discovery
1
2Generate an atomic list with no choices, no target count, and no hints about where changes occur.
Ground-truth differences
VDiff-Bench-Gen
Open-ended identification tests both correctness and completeness. Fluent outputs can still omit subtle changes, reverse their direction, or introduce unsupported claims.
Open-ended performance
Micro-F1 rewards correct coverage while exact-set accuracy requires every annotated difference and no unsupported claims.
Complete results
Reported benchmark scores from the paper.
| # | Model | Access | Precision | Recall | Micro-F1 | Pair macro-F1 | Exact set |
|---|
All values are percentages. Exact set requires complete recovery of the annotated difference set.
Evaluating open-ended outputs
Instead of asking an MLLM for one opaque score, VDiff-Judge decomposes each response into atomic claims, checks targeted image evidence, and retains an inspectable matching trace.
Separate the generated response into atomic differences.
Compare object, attribute, location, and direction.
Use global measurements, grounding, and targeted crops.
Accept supported correspondences and retain their traces.
Compute precision, recall, F1, and exact-set accuracy.
VDiff-Bench-MCQ examples
VDiff-Bench-MCQ
VDiff-Bench-MCQ contains 1,756 four-way questions across ten semantic, textual, and low-level categories. Each question pairs one verified difference with two hard alternatives and a no-difference distractor, enabling exact, judge-free scoring. The controlled format reveals whether models can distinguish the observed change from visually plausible competitors.
Why VDiff-Bench
Prior benchmarks cover surveillance, rendered, and other paired images but rely on open-ended outputs scored by caption metrics or learned judges. VDiff-Bench spans real, edited, rendered, and 2D puzzle images, covering semantic, OCR/text, photometric, noise/resolution, and texture changes. Models must distinguish the observed change from plausible alternatives, enabling exact, judge-free scoring.
| Benchmark | Pairs | Image regime | Target-change coverage | Evaluation design | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Real | Edited | Rendered | 2D puzzle | Semantic | OCR/text | Global photo. | Noise/res. | Texture | Hard alternatives | Exact scoring | ||
| Spot-the-Diff | 13,192 | ✓ | × | × | × | ✓ | × | × | × | × | × | × |
| CLEVR-Change | 79,606 | × | × | ✓ | × | ✓ | × | × | × | ✓ | × | × |
| OmniDiff | 15,598 | ✓ | × | ✓ | × | ✓ | ✓ | ✓ | × | × | × | × |
| DiffCap-Bench | 1,075 | ✓ | ✓ | ✓ | × | ✓ | ✓ | ✓ | ✓ | ✓ | × | × |
| VDiff-Bench-MCQ (ours) | 1,543 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Checks mark explicitly covered dimensions. “Global photo.” includes whole-image color and illumination; “Noise/res.” includes noise and resolution degradation. Hard alternatives are plausible competing changes; exact scoring uses direct answer-key matching without caption metrics or learned judges.
Dataset statistics
Hover or focus a segment for its definition, group, and question count.
VDiff-Bench-MCQ performance
Overall accuracy and low-level accuracy on whole-image color, noise, texture, and illumination changes.
Category explorer
Full results
Accuracy (%). Uniform four-way guessing yields 25.0%; excluding the always-false no-difference option yields 33.3%.