VDiff-Bench Diagnosing and Improving Fine-Grained Visual Difference Understanding

Authors

Yixin Wan Tianle Zheng Kai-Wei Chang

University of California, Los Angeles

2benchmark tracks
1,756MCQ questions
300Gen image pairs
12evaluated MLLMs

Two complementary tracks

Recognition with choices. Discovery without them.

VDiff-Bench separates controlled identification from open-ended discovery, exposing whether a model can recognize a difference and whether it can find every difference independently.

VDiff-Bench-MCQ

Controlled identification
First farm-scene image1
Second farm-scene image2
01

Which difference is real?

Select one verified difference from two plausible alternatives and a no-difference distractor.

  1. AThe white duck faces left in Image 1 but faces forward in Image 2.
  2. BThe pig's tail points downward in Image 1 but curls upward in Image 2.Ground truth
  3. CThe horse's tail is brown in Image 1 but black in Image 2.
  4. DNo difference between Image 1 and Image 2.

VDiff-Bench-Gen

Open-ended discovery
First underwater puzzle image1
Second underwater puzzle image2
02

Identify every visible difference.

Generate an atomic list with no choices, no target count, and no hints about where changes occur.

Ground-truth differences

  • 1The smallest bubble beside the jellyfish disappears.
  • 2A piece of seagrass in the upper-left disappears.
  • 3A purple rock in the upper-right disappears.
  • 4A water ripple in the center-right disappears.

VDiff-Bench-Gen

Leading models still miss most complete difference sets

Open-ended identification tests both correctness and completeness. Fluent outputs can still omit subtle changes, reverse their direction, or introduce unsupported claims.

300challenging image pairs
46.8%best reported micro-F1
17.3%best exact-set accuracy
01/03

Open-ended performance

Coverage remains low even for proprietary models

Micro-F1 rewards correct coverage while exact-set accuracy requires every annotated difference and no unsupported claims.

Micro-F1 Exact-set accuracy

Complete results

VDiff-Bench-Gen leaderboard

Reported benchmark scores from the paper.

VDiff-Bench-Gen precision, recall, F1, and exact-set accuracy
#ModelAccessPrecisionRecallMicro-F1Pair macro-F1Exact set

All values are percentages. Exact set requires complete recovery of the annotated difference set.

Evaluating open-ended outputs

VDiff-Judge verifies claims against visual evidence

Instead of asking an MLLM for one opaque score, VDiff-Judge decomposes each response into atomic claims, checks targeted image evidence, and retains an inspectable matching trace.

97%human agreementCohen's κ = 0.937
1Parse claims

Separate the generated response into atomic differences.

2Propose matches

Compare object, attribute, location, and direction.

3Inspect evidence

Use global measurements, grounding, and targeted crops.

4Verify one-to-one

Accept supported correspondences and retain their traces.

5Score the set

Compute precision, recall, F1, and exact-set accuracy.

VDiff-Bench-MCQ examples

Controlled identification across 10 change categories

VDiff-Bench-MCQ

Plausible alternatives isolate fine-grained recognition

VDiff-Bench-MCQ contains 1,756 four-way questions across ten semantic, textual, and low-level categories. Each question pairs one verified difference with two hard alternatives and a no-difference distractor, enabling exact, judge-free scoring. The controlled format reveals whether models can distinguish the observed change from visually plausible competitors.

Qualitative comparison of VDiff-Bench with CLEVR-Change, Spot-the-Diff, and OmniDiff
Figure 1. VDiff-Bench compared with prior visual-difference benchmarks and representative model responses.

Why VDiff-Bench

Controlled visual verification

Prior benchmarks cover surveillance, rendered, and other paired images but rely on open-ended outputs scored by caption metrics or learned judges. VDiff-Bench spans real, edited, rendered, and 2D puzzle images, covering semantic, OCR/text, photometric, noise/resolution, and texture changes. Models must distinguish the observed change from plausible alternatives, enabling exact, judge-free scoring.

Coverage and evaluation design of visual difference identification benchmarks
Benchmark Pairs Image regime Target-change coverage Evaluation design
RealEditedRendered2D puzzle SemanticOCR/textGlobal photo.Noise/res.Texture Hard alternativesExact scoring
Spot-the-Diff13,192✓×××✓××××××
CLEVR-Change79,606××✓×✓×××✓××
OmniDiff15,598✓×✓×✓✓✓××××
DiffCap-Bench1,075✓✓✓×✓✓✓✓✓××
VDiff-Bench-MCQ (ours)1,543✓✓✓✓✓✓✓✓✓✓✓

Checks mark explicitly covered dimensions. “Global photo.” includes whole-image color and illumination; “Noise/res.” includes noise and resolution degradation. Hard alternatives are plausible competing changes; exact scoring uses direct answer-key matching without caption metrics or learned judges.

Dataset statistics

Ten complementary change categories

Hover or focus a segment for its definition, group, and question count.

Semantic or textualLow-level

VDiff-Bench-MCQ performance

Strong averages can hide perceptual gaps

Overall accuracy and low-level accuracy on whole-image color, noise, texture, and illumination changes.

Overall accuracy Low-level accuracy Guessing baselines

Category explorer

Compare models on a specific change

Category accuracyGuessing baselines

Full results

VDiff-Bench-MCQ leaderboard

Model accuracy by VDiff-Bench change category

Accuracy (%). Uniform four-way guessing yields 25.0%; excluding the always-false no-difference option yields 33.3%.