Vision and Multimodality · ICML 2025

Visual Graph Arena Deep Report

Models See Lines, but Do Not Reliably Recover the Graph Behind Them

ICML 2025 / PMLR 267 2026-08-25 Visual reasoning · Graph invariance · MLLM evaluation

VGA shows models lack a layout-robust pixel-to-graph mapping; test leakage and entangled perception/reasoning leave pseudo-intelligence a hypothesis rather than an established explanation.

Date: 2026-08-25
Today's field: Vision and Multimodality
Core paper: Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models
Publication: ICML 2025, PMLR 267:2081–2113
Random-selection record: 24 candidates; executed shuf -i 1-24 -n 1 and selected entry 7. Official full text, project page, and data were accessible; no redraw.

Core positioning: VGA changes layouts of identical graph structures rather than merely asking graph questions on images, testing layout-independent relational invariants. Humans score 88.2%–100%, while visual models approach chance on isomorphism; the best general visual model reaches only 36.3% on largest chordless cycles.

0. Link verification table

All links were accessed on 2026-08-25. VGA's page offers downloads/loader examples, but no full training/evaluation repository was found, so loader snippets are not labeled complete reproduction code.

The paper Conference page / formal publication arXiv (version date) Code repository Datasets / weights Project page / video
Visual Graph Arena (core) ICML 2025 / PMLR; OpenReview 2506.06242v1, 2025-06-06 Full training code not found; Webpage loader example Direct ZIP download; Zenodo DOI Project page; video not found
CLEVR CVPR 2017 1612.06890v1, 2016-12-20 Data-generation code CLEVR data page Same data page; video not found
FigureQA Official conference page not found 1710.07300v2, 2018-02-22 Official repository; baseline Download instructions in repository Project page not found
Visual Concept-Metaconcept Learning NeurIPS 2019 2002.01464v1, 2020-02-04 Not found Uses CLEVR / GQA / CUB Project page not found
VisionGraph Official conference page not found 2405.04950v1, 2024-05-08 Official repository Benchmark in repository Project page not found

0.1 Candidate pool (24 papers)

All candidates are ICML 2025 main-conference papers within the preferred past 12 months; MMedPO and medical preference optimization, covered deeply on 2026-08-18, are excluded.

  1. GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model
  2. Vision Graph Prompting via Semantic Low-Rank Decomposition
  3. Direct Motion Models for Assessing Generated Videos
  4. Aligning Multimodal Representations through an Information Bottleneck
  5. Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes
  6. FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
  7. Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models
  8. Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in VLMs
  9. SMART-PC: Skeletal Model Adaptation for Robust Test-Time Training in Point Clouds
  10. Impossible Videos
  11. Whitened CLIP as a Likelihood Surrogate of Images and Captions
  12. Rethinking Point Cloud Data Augmentation: Topologically Consistent Deformation
  13. Point Cloud Dataset Distillation
  14. DEFAME: Dynamic Evidence-based Fact-checking with Multimodal Experts
  15. From Local Details to Global Context: Advancing VLMs with Attention-Based Selection
  16. Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts
  17. From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation
  18. EFDTR: Learnable Elliptical Fourier Descriptor Transformer for Instance Segmentation
  19. AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment
  20. CtrlSynth: Controllable Image Text Synthesis for Data-Efficient Multimodal Learning
  21. Open-Det: An Efficient Learning Framework for Open-Ended Detection
  22. SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
  23. LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in MLLMs
  24. A Closer Look at Multimodal Representation Collapse

1. In-depth analysis of the core paper

1.1 Motivation: separate visual parsing from structural invariants

Traditional VQA combines recognition, language understanding, commonsense, and reasoning in one score, obscuring whether errors reflect perception, comprehension, reasoning, or bias. VGA controls tiny undirected graphs with 8–9 clearly drawn nodes/edges, inspectable by humans. Identical abstract graphs can look very different under random, Kamada–Kawai, or planar layouts. Labels depend only on structure, making appearance a controlled intervention.

The paper calls the target conceptualization: recognizing/manipulating unchanged concepts despite changed representations. This is narrower than general OOD. Colors/backgrounds/capture domains may change without relations changing; VGA changes node coordinates, edge lengths, angles, crossings, and silhouettes while preserving adjacency or graph properties. Ideal models ignore surface layouts and recover equivalent relational representations.

The quantitative gap is direct. Table 2's human scores across eight columns are 88.2%–100%; visual models on both isomorphism tasks are almost all marked FAIL, except SigLIP on easy isomorphism at 54.4%, only 4.4pp above binary chance. On largest chordless cycles, strongest ConvNeXt scores only 36.3%, trailing humans' 88.2% by 51.9 pp. This is not a small remaining gap: structural invariants do not reliably emerge from pixel representations.

Flowcharts, chemical structures, circuits, electrical diagrams, molecular skeletons, and scientific schematics all admit different drawings of identical relations. Models relying on silhouettes/textures may excel on familiar templates yet fail with new layouts, software, or standards. VGA offers a minimal counterexample rather than simulating every diagram: failure across layouts of 8–9-node graphs challenges claims of general visual structural understanding.

1.2 Narrative: controlled interventions reveal behavioral anomalies

The paper builds a diagnostic chain rather than a new network.

First: choose structurally explicit, enumerable visual objects. Images come from connected 8/9-node graphs; Appendix B reports 11,117 eight-node and 261,080 nine-node graphs. Graph algorithms supply deterministic labels, avoiding subjective annotation.

Second: make labels structural and separate train/test layouts. Six tasks cover isomorphism, paths, and cycles, each with two difficulties/forms. In principle, memorizing training-layout shapes should not suffice; cross-layout invariants are required.

Third: test three levels. Humans provide a cognitive reference; ImageNet-pretrained ViT/Swin/ConvNeXt plus SigLIP/DINOv2 are fine-tuned to test transfer after substantial supervision; closed MLLMs answer images directly to test zero-shot generality.

Fourth: examine error structure beyond scores. GPT-o1 scores 55% overall on shortest paths, yet Table 3 gives only 28% for length 1 versus 69% for length 2. This Easier-Worse Anomaly questions whether adjacency is understood: the easiest class should not trail longer paths by 41pp. Hamiltonian-cycle 66% also largely exploits leaf-node detection; removing leaf cases approaches chance.

To isolate visual conceptualization, MLLMs are explicitly instructed not to use code. This resembles human visual intuition tests but excludes practical modern agents that transcribe adjacency then call deterministic algorithms. VGA diagnoses direct vision-to-structure reasoning rather than the best attainable tool-augmented performance.

1.3 Formalization: what VGA actually measures

Let abstract graph (G=(V,E)), layout algorithm (ellL), and renderer (R) produce:

[ x_{G,}=R(G;). ]

Labels depend only on structure:

[ y=(G), ]

where ( au) denotes isomorphism, Hamiltonian path/cycle existence, shortest-path length between marked nodes, or largest chordless-cycle length. Ideal reasoner (f_) satisfies layout invariance:

[ f_(R(G; 1))=f(R(G;_2))=(G), _1,_2L. ]

For training layouts (mathcal L_{tr}) and unseen test layouts (mathcal L_{te}), the intended layout-generalization gap is:

[ {layout}=(L{tr})- (L_{te}). ]

Except for shortest-path multilayout tests, the paper lacks systematic paired in-layout/cross-layout results for identical models and underlying graph distributions, so most tasks' (Delta_{layout}) is not computable. Reported cross-layout accuracy is not a full invariance curve.

Six labels can be formalized as:

  • Isomorphism: (y=[: A_2=P_A_1P_^]); hard negatives share positive degree sequences, excluding simple degree shortcuts.
  • Hamiltonian path: (y=[(v_1,,v_{|V|}), (v_i,v_{i+1})E]).
  • Shortest paths: (y=d_G(s,t){1,2,3,4}).
  • Hamiltonian cycles: add ((v_{|V|},v_1)E) to Hamiltonian-path conditions.
  • Largest chordless cycle: (y=_{CG}|C|{3,4,5,6}), with no chords between nonadjacent cycle nodes.

Mechanistically, success should follow:

Pixel image
  ↓ Node detection + edge tracing
Explicit/implicit adjacency Â
  ↓ Permutation/layout-invariant relational operations
Graph property τ(Â)
  ↓
Answer

VGA observes only final answers and uses 3 adjacency-transcription examples in Appendix D. Without node/edge parsing accuracy across all samples, errors may arise from unread edges or failed reasoning after correct parsing.

1.4 Dataset and implementation

Mechanism / dimension Design choice Motivation Key reported configuration
Graph size Connected graphs, 8–9 nodes Human-inspectable and scalable to many samples Original set: 11,117 eight-node; 261,080 nine-node graphs, Appendix B
Render 700×700 original canvas; random / Kamada–Kawai / planar Change appearance while preserving structural labels Appendix B
Task Two each: isomorphism, paths, cycles Binary/four-class tasks; local/global structures Main text §3
Class balance Balanced binary classes and four path/cycle lengths Reduce label-prior shortcuts Main text §3, Appendix B
Visual models ViT-B, Swin-T-B, ConvNeXt-B, SigLIP-B, DINOv2-B Compare convolution, supervised Transformers, vision-language, and self-supervised pretraining Main text §4
Fine-tuning Adam, LR (10^{-4}), batch 32, 10 epochs, CrossEntropy Task-supervised adaptation Appendix C, page 12
Input size 384×384 Match pretrained input settings Appendix C
Hardware 2× NVIDIA TITAN RTX Visual-model training Appendix C
Model selection Evaluate test sets each epoch; select the best test epoch Original implementation Appendix C; test-leakage risk
MLLM 100 random samples per task; code prohibited Direct visual reasoning Main text §4, Appendix E
humans 15 engineering students/staff; 24 questions, 4 per task Human reference Main text page 6

Six-task size matrix

Concept Task Training samples Test samples Category Training → test layouts Source
Isomorphism Easy 140,000 15,718 2 paired Kawai/planar; graph split Table 1, Appendix B.1
Isomorphism Hard degree-equivalent 127,374 14,298 2 Paired Kawai/planar; hard negatives share degree sequences Table 1, Appendix B.1
Path Hamiltonian path 25,000 2,480 2 Kawai → planar Table 1, §3.2
Path Shortest path 80,000 8,672 4 Table 2 footnote/directory diagram: random→random/Kawai/planar; Appendix B.2 text instead says Kawai→planar Table 1, Table 2 footnote, Appendix B.2
Cycle Hamiltonian cycle 69,935 7,740 2 Kawai → planar Table 1, §3.3
Cycle Biggest chordless cycle 80,000 6,484 4 Kawai → planar Table 1, §3.3

Table 1's total training size is 522,309, and summing the listed task test sets gives 55,392. Shortest paths actually have three test layouts; the appendix directory diagram also lists separate Kawai/planar directories for Hamiltonian paths/cycles and chordless cycles. Whether Table 1's testing counts are per-layout or combined is unclear, preventing a direct total image-file count.

1.5 Main results: complete verifiable matrix

Values transcribe Table 2; FAIL borrows Long Range Arena terminology for near-chance performance without exact values or a common threshold. Binary chance is 50%; four-class chance 25%. Dashes mean unreported model/layout combinations.

Method Iso-Easy Iso-Hard Ham. Path Shortest-Random Shortest-Kawai Shortest-Planar Ham. Cycle Chordless Source
Human 95.0 91.6 96.6 98.3 100.0 100.0 93.3 88.2 Table 2, row 1
ViT Base FAIL FAIL 57.0 67.4 66.7 64.4 67.8 32.3 Table 2, row 2
Swin-T Base FAIL FAIL 65.8 68.6 65.7 65.4 71.1 34.4 Table 2, row 3
ConvNeXt Base FAIL FAIL 72.9 73.3 82.4 73.3 74.9 36.3 Table 2, row 4
SigLIP Base 54.4 FAIL 59.5 25.2 — — 63.5 28.0 Table 2, row 5
DINOv2 Base FAIL FAIL 56.8 36.4 — — FAIL 31.1 Table 2, row 6
GPT-o1 FAIL FAIL FAIL 55.0 — — 66.0 FAIL Table 2, row 7
GPT-4o FAIL FAIL FAIL FAIL — — FAIL FAIL Table 2, row 8
Claude 3.5 Sonnet FAIL FAIL FAIL FAIL — — FAIL FAIL Table 2, row 9

Recomputed means over six exact-valued non-isomorphism columns for ViT/Swin/ConvNeXt are 59.27%, 61.83%, 68.85%; ConvNeXt exceeds Swin by 7.02 pp and ViT by 9.58 pp. This descriptively favors convolution on these thin-line diagrams, but without seeds, error bars, or significance tests, architecture cannot robustly explain the difference.

Human gaps over each strongest visual model: easy isomorphism 40.6pp, Hamiltonian path 23.7pp, shortest-random 25.0pp, shortest-Kawai 17.6pp, shortest-planar 26.7pp, Hamiltonian cycle 18.4pp, chordless cycle 51.9pp. The 51.9pp chordless-cycle gap is the strongest single finding: ConvNeXt's 36.3% is only 11.3pp above 25% chance, versus humans' 88.2%.

GPT-o1 error matrix

Ground truth  Prediction 1 2 3 4 Class accuracy Source
Length 1 28.0 67.0 5.0 0.0 28.0 Table 3, row 1
Length 2 18.0 69.0 13.0 0.0 69.0 Table 3, row 2
Length 3 0.0 40.0 60.0 0.0 60.0 Table 3, row 3
Length 4 0.0 5.0 40.0 55.0 55.0 Table 3, row 4

The clearest failure concerns adjacent nodes, not long paths: 67% of length-1 examples are predicted as 2, motivating Easier-Worse Anomaly. Yet Table 3 says 28%/69%, §5.3 says 29%/70%; Hamiltonian-cycle Table 2 says 66.0%, its note 66.6%, and text 67%. Apparent rounding still exposes inconsistent reporting.

1.6 Ablations, sensitivity, and significance

Ablation Studies

The paper reports no standard model ablations. Missing progressive comparisons include:

Needed ablation Reported? Unanswered question
Same underlying graphs, layouts alone changed Partly intended by task design; no paired per-graph results How much error comes from layout change?
Same-layout in-domain versus cross-layout OOD Not systematically reported except shortest paths (Delta_{layout}) cannot be computed
Image → oracle adjacency → solver Not reported Cannot separate parsing from graph-reasoning errors
Single-factor position/crossing/line-width/node-size sweeps Not reported Which visual factors cause failure?
Data-scaling curves Not reported Are 27k–140k examples saturated, and would more help?
Remove ImageNet pretraining / freeze backbones Not reported Cannot distinguish pretraining benefits from negative transfer
Multiple seeds and epoch-selection protocols Not reported Is ConvNeXt's advantage stable?

Appendix D adjacency transcription uses only 3 images, qualitative cases rather than sufficient perception ablations. All closed MLLMs struggle, with Claude 3.5 Sonnet closest on node/edge counts, suggesting some graph errors originate in visual extraction.

Hyperparameter sensitivity

Only one configuration is given: LR (10^{-4}), batch 32, 10 epochs, 384×384, Adam, 2×TITAN RTX. No sensitivity curves are reported for LR, resolution, augmentation, epochs, weight decay, or node count. Neither an 8–9-node transition threshold nor benefits of raising resolution from 384 to 768 can be established.

A possible task transition warrants testing: exact chordless-cycle results are only 28.0%–36.3%, near 25% chance, while binary Hamiltonian-cycle visual scores reach 63.5%–74.9%. This could reflect global-cycle enumeration or leaf-node shortcuts. Without stratifying leaf counts, edge density, and crossings, the cause remains unknown.

Statistical significance

Visual models report one best test accuracy without deviations; MLLMs use 100 samples/task without CIs; humans number only 15 with 4 questions/task. No p-values, CIs, bootstrap, paired tests, or effect sizes appear. Worse, Appendix C explicitly selects the best test-performing epoch, introducing optimistic bias through 10 test-set model-selection opportunities rather than a sealed evaluation.

1.7 Contributions, reuse, and transfer

VGA contributes a benchmark-design principle beyond six graph questions: hold concepts fixed while controllably intervening on representations. Typical visual OOD tests jointly alter backgrounds, styles, frequencies, and sources, obscuring causes. VGA aims to change only layout, making conceptual invariance testable.

Reusable designs include:

  1. paired renderer intervention: render one structure in multiple layouts as counterfactual image pairs, transferable to molecules, circuits, flowcharts, transport networks, and UI wireframes.
  2. hard negative by invariant matching: isomorphism hard negatives preserve degree sequences, forcing models beyond node/edge counts and degree statistics.
  3. Error-monotonicity audits: assess whether easier instances perform worse, not just overall accuracy; transferable to mathematics, spatial relations, counting, and tool-use benchmarks.
  4. Minimal human–model capability gaps: use small tasks humans solve without special training, avoiding confusion between expertise and perception gaps.

Replacing structures with circuit netlists, chemical graphs, or workflow dependencies tests recovery across drawing conventions. Next-generation benchmarks should include pixel-only, oracle-adjacency, and tool-enabled tracks for perception, reasoning, and orchestration separately.

1.8 Limitations

Author-stated or explicitly acknowledged limitations

  1. Limited task domain. Conclusion/Future Work (p.8) notes small graphs only and proposes chemical structures/circuits; cross-domain conceptualization is unvalidated.
  2. Graph transcription itself is difficult. Appendix D (p.17) acknowledges GPT-4o/o1 and Claude 3/3.5 cannot reliably produce correct adjacency matrices, identifying visual parsing as a primary bottleneck.
  3. Imitative shortcuts caused by probabilistic training remain a hypothesis, not an established mechanism. §5 links pseudo-intelligence to training paradigms and suggests RL, without experiments validating this explanation.

Additional critique (10 independent observations)

  1. Experimental design | Test leakage. Appendix C evaluates each epoch on test data and selects maximum test accuracy, effectively making 10 implicit selection attempts per model. Table 2 is not a strictly held-out estimate; differences may be optimistically inflated.
  2. Argument gap | Perception and graph reasoning are entangled. Appendix D already shows adjacency-transcription failures, but no same-model oracle-adjacency evaluation separates them. Apparent absent conceptualization may be line/node-reading errors; final accuracy cannot locate reasoning deficits.
  3. Experimental design | Layout and graph-distribution shifts are confounded. Appendix B.2 puts planar shortest-path graphs in test and remaining nonplanar graphs in train before changing drawing layouts. Underlying graph families may also shift, so the entire gap cannot be attributed to representations.
  4. Metric validity |FAIL hides effect sizes. Table 2 uses FAIL instead of over a dozen exact values; with chance at 50% or 25%, readers cannot distinguish 49.9%, 25.1%, or below-chance results, compute means/CIs, or rank fairly.
  5. Statistical design | Small, unclear human protocol. 15 people ×4 questions/task yield at most 60 judgments per task, yet shortest paths have random/Kawai/planar columns. How 24 questions cover layouts, repeated graphs, or variants is unspecified, leaving denominators unauditable.
  6. Reproducibility | Missing closed-model versions/settings. GPT-4o/o1, Claude 3.5, and Gemini evolve, but exact snapshots, API dates, temperatures, max tokens, retries, and refusal handling are absent. The 100-sample results cannot be stably reproduced.
  7. Overclaiming | Pseudo-intelligence is not the only explanation. GPT-o1's length-1 28% versus length-2 69% clearly shows nonmonotonicity, but square markers occluding edges, ambiguous contacts, or prompt parsing could explain it. Perceptual artifacts remain unexcluded without visibility controls.
  8. Missing comparison | No tool-augmented track. VisionGraph demonstrates description→adjacency→code/interpreter DPR, while VGA prohibits code. It shows direct visual answering is hard, not that practical agents cannot solve tasks. Compare pixel-only, oracle graphs, OCR/edge detectors, and Python solvers on the same 100 samples.
  9. Failing subtask | Large supervision remains near chance. Chordless cycles have 80,000 training examples, yet ConvNeXt reaches only 36.3%, 11.3pp above chance. More supervision alone is unsupported as a remedy; scaling curves should distinguish data scarcity from representation/optimization mismatch.
  10. Reporting consistency | Drifting numbers and settings. GPT-o1 shortest-path Table 3 gives 28/69 versus §5.3's 29/70; Hamiltonian-cycle Table 2 gives 66.0, its note 66.6, text 67. Shortest-path training layout is random in the Table 2 footnote/directory diagram but Kamada–Kawai in Appendix B.2. Even if numbers reflect rounding, layout discrepancies alter reproduction and require code/raw counts to resolve.

2. Related-work retrospective: debiased VQA to layout counterfactuals

2.1 CLEVR — 2017, CVPR

What it solved. CLEVR uses procedural scenes, functional question templates, and traceable reasoning programs for diagnostic recognition/counting/comparison/spatial/compositional tasks. It reports 100,000 rendered images and approximately 1 million questions, 853,000 unique, demonstrating synthetic data's value for controlled bias/reasoning rather than realism.

What remained. CLEVR appearances, styles, and relation grammars are relatively fixed. CoGenT changes color–shape combinations, not systematically redrawing identical relations in entirely different geometries.

VGA's response. VGA retains procedural generation, deterministic labels, and synthetic control, replacing object-attribute interventions with graph layouts. Fixed ( au(G)) with changed coordinates/edge geometry tests relationships rather than drawing conventions.

Design inheritance. CLEVR's functional programs/decomposed diagnosis become six atomic graph tasks, but VGA lacks executable per-sample reasoning traces, weakening attribution.

2.2 FigureQA — 2017/2018, preprint

What it solved. FigureQA combines synthetic bars, lines, and pies with relational QA: over 100,000 images and 1 million pairs covering inequalities, extrema, trends, and intersections, extending reasoning into scientific charts.

What remained. It mainly asks visible data relationships; stable reasoning across themes, layouts, or libraries is not central. Models may still learn chart templates, colors, and position statistics.

VGA's response. VGA uses abstract graphs without text/numeric axes to avoid OCR/value-reading dominance, explicitly changing geometry through Kawai/random/planar layouts to separate structure from appearance.

Design inheritance. Programmatically generated scientific visuals and automatically verifiable answers carry over; layout becomes OOD intervention, at the cost of chart-type/question diversity.

2.3 Visual Concept-Metaconcept Learning — 2019, NeurIPS

What it solved. VCML learns visual concepts such as red/cube plus metaconcepts such as synonym/same-kind/hypernym, executing neural-symbolic programs. Learnable concept-relation operators are tested for generalization on CLEVR/GQA/CUB.

What remained. VCML assumes object proposals, semantic parsing, and DSLs, reasoning in structured concept embeddings. It does not establish whether end-to-end pixel models recover invariants without explicit scene graphs.

VGA's response. VGA hides intermediate structure, offering only pixels and short questions to test spontaneous relational representations. Even on 8–9-node graphs, computable human–model gaps remain 18.4–51.9pp.

Design inheritance. Concepts as relational structures operable across instances/representations carry over, but final accuracy without explicit operators/program traces gives VGA coarser diagnostics.

2.4 VisionGraph — 2024, arXiv

What it solved. VisionGraph poses 8 graph-task types across 5,902 problems, 3 questions each, totaling 17,706 QA pairs. It tests nodes/edges, connectivity, cycles, topological sorting, shortest paths, max flow, matching, Hamilton paths, and GNN updates. Description–Program–Reasoning first builds descriptions/adjacency, then algorithms/code, optionally calling Python.

What remained. Its main axes are task type and node-count difficulty, not layout invariance; larger graphs may be crowded. Combining perception, program generation, and tools also obscures human-like conceptualization alone.

VGA's response. VGA limits graphs to 8–9 inspectable nodes, intervenes on train/test layouts, and forbids code to isolate end-to-end visual abstraction.

Design inheritance. VGA inherits graph-image benchmarks, adding large supervised sets (25,000–140,000 per task) and humans. DPR highlights the missing practical route: explicit adjacency/tools can compensate for weak end-to-end intuition.

Humans versus ConvNeXt: cross-layout gaps remain 17.6–51.9pp

Only six Table 2 columns with exact values for both are plotted; isomorphism marked FAIL is not estimated from pixels.

Human ConvNeXt Base
Human and ConvNeXt Base accuracy across six VGA tasks Humans score 96.6, 98.3, 100, 100, 93.3, and 88.2 on Hamiltonian paths, shortest paths in random/Kawai/planar layouts, Hamiltonian cycles, and chordless cycles; ConvNeXt scores 72.9, 73.3, 82.4, 73.3, 74.9, and 36.3. 020406080100% Ham. Path SP Random SP Kawai SP Planar Ham. Cycle Chordless
Unit: accuracy (%). Source: Table 2. Bars share one 0–100% horizontal axis; exact values/source rows appear in the body matrix.

3. Technical evolution

2017 CLEVR [synthetic scenes; procedural questions; reduced language/label bias]
   ├── 2017/18 FigureQA [scientific charts; automatic QA; relational comparisons]
   └── 2019 VCML [concepts + metaconcepts; explicit operators; neural-symbolic programs]
                  │
                  ├── Route A: explicit structure before reasoning
                  │      2024 VisionGraph [nodes/edges → DPR → code/tools]
                  │
                  └── Route B: diagnose end-to-end structural invariants
                         2025 VGA [same graph × multiple layouts; no code]
                                  ├─ Isomorphism near chance
                                  ├─ Length-1 trails length-2 by 41pp
                                  └─ Gap: perception/reasoning not separated

The driver shifts three times: from detecting guesses based on language bias, to composing concepts/relations, to preserving relations across visual realizations. VGA turns layout from nuisance into an experimental variable. The next step may reunite routes: pixel-only tracks for intuitive conceptualization and structured/tool tracks for practical completion, with structural-recovery losses between them reported.

3.1 Cross-paper comparison

Work Publication date Conference/publication Core contribution Scale / main metrics Relation to VGA
CLEVR 2017.07 CVPR Procedural synthetic visual-reasoning diagnosis 100k images; approximately 1M questions Controlled generation and atomic-reasoning paradigm
FigureQA 2018.02 (v2) arXiv Scientific-chart relational QA >100k images; >1M QA Extends synthetic reasoning into chart vision
VCML 2019.12 NeurIPS Joint visual-concept/metaconcept learning CLEVR/GQA/CUB Concepts as operable relations rather than categories
VisionGraph 2024.05 arXiv 8 visual graph-task classes and DPR agents 5,902 problems; 17,706 QA Closest predecessor; emphasizes tools rather than layout invariance
VGA 2025.07 ICML Six-task large cross-layout image benchmark 522,309 training examples; Table 2 humans 88.2–100% Layout as a controlled counterfactual variable

4. Open challenges and actionable research opportunities

4.1 Theory and evaluation

  1. Decompose conceptualization into identifiable errors. For every graph, measure node F1, edge F1, graph-edit distance, and final-property accuracy; run the same reasoner with oracle adjacency. Compare pixel→answer, pixel→adjacency→answer, oracle adjacency→answer to quantify perceptual versus reasoning errors.
  2. Estimate layout-generalization gaps rigorously. Fix underlying graphs and generate random/Kawai/planar views. Use graph-stratified splits with no identity overlap and report a 3×3 train-layout→test-layout matrix rather than only Kawai→planar.
  3. Joint invariance–accuracy metrics. Beyond accuracy, report agreement across three views: [ C(G)=_{_i_j} [f(R(G;_i))=f(R(G;_j))]. ] Distinguish consistently wrong predictions from inconsistent ones.

4.2 Engineering and applications

  1. Formal reproduction without test leakage. Split train/validation/test by graph identity, choose epochs on validation, and test once. Report 5-seed means±std and paired bootstrap beside original test-selected results to quantify optimism.
  2. Resolution and visual-degradation sweeps. Compare ConvNeXt, ViT, and modern VLMs at 224/384/768/1024 resolution, 1–5px lines, varied antialiasing, node radii, and crossings. Identify structural-reading thresholds rather than merely maximizing scores.
  3. Equal-budget architecture comparisons. Match parameter counts, pretraining data, resolution, and steps across ConvNeXt, ViT, Swin, and graph-aware encoders. Otherwise Table 2's 9.58pp mean gap cannot be attributed to convolutional bias.
  4. Tool-augmented head-to-head tests. On the same 100 samples/API snapshot, compare direct answers, CoT, adjacency transcription, and NetworkX calls, reporting accuracy, tokens, latency, and adjacency-edit distance.

4.3 Emerging directions

  1. From static graphs to relational constancy in video. Move nodes continuously with edges fixed, comparing single-frame VLMs, video encoders, and explicit trackers. Test separation of layout motion from topology changes.
  2. Cross-domain isomorphic transfer. Render identical graphs as circuits, molecules, flowcharts, and social networks; train on three domains and test the fourth, a stronger conceptual-transfer test than three within-domain layouts.
  3. Generation–verification loops. Have VLMs generate adjacency, rerender with differentiable renderers, and align structure to input. Compare reconstruction supervision, contrastive invariance, and classification-only losses with matched data/compute.
  4. Safety and reliability. Add thin edges, occlusions, or false crossings to critical engineering diagrams and measure graph-edit robustness. Require abstention under high adjacency uncertainty rather than fluent erroneous graph explanations.

5. Other recent work worth watching

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (2025.07, ICML)
Analyzes VLM spatial-reasoning failures through attention focus, complementing VGA's node/edge bottlenecks mechanistically.
PMLR

Vision Graph Prompting via Semantic Low-Rank Decomposition (2025.07, ICML)
Explicitly graphs visual relations and prompts with low-rank semantic structures, providing structural priors rather than merely more end-to-end data.
PMLR

From Local Details to Global Context: Advancing VLMs with Attention-Based Selection (2025.07, ICML)
Studies local-detail selection and global-context integration, the two scales covered by thin-edge reading and whole-graph cycle reasoning.
PMLR

A Closer Look at Multimodal Representation Collapse (2025.07, ICML)
Diagnoses multimodal representation collapse, asking whether layout invariance accidentally discards structural information.
PMLR

Contrastive Localized Language-Image Pre-Training (2025.07, ICML)
Moves image–text alignment from global matching to local grounding; node/edge grounding is prerequisite to VGA adjacency recovery.
PMLR

Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation (2025.07, ICML)
Hierarchical compression of very long videos suggests dynamic VGA extensions must control both relation preservation and token-compression loss.
PMLR

LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Models (2025.07, ICML)
Scales long visual inputs at system level, suitable for large circuit/flowchart benchmarks, though seeing more does not guarantee correct structure recovery.
PMLR

6. Final assessment

VGA offers valuable diagnosis through three reinforcing findings: near-chance isomorphism, a 51.9pp human–ConvNeXt chordless-cycle gap, and GPT-o1's 28% on simplest length-1 paths versus 69% at length 2. These reject optimistic claims that current visual models reliably understand small-graph structure.

But interpreting this as conceptualization/pseudo-intelligence moves too quickly. Test-based epoch selection leaks information, FAIL hides exact values, human/closed-model protocols are incomplete, and no oracle-adjacency control distinguishes unread edges from failed reasoning. The stronger conclusion is: Current models lack a layout-robust end-to-end pixel-to-graph-answer mapping. VGA has not causally separated perception, structure recovery, algorithmic reasoning, and their interfaces.


Index status: updated ml-report-index.md.
A candid note: no complete training repository, exact accuracies corresponding to FAIL, multiseed visual results, sensitivity analyses, or strictly held-out tests are available; none are invented here. HTML uses a single-file Warm Editorial Report template and passes structural validation. The runtime lacked executable Chromium under /usr/bin/chromium or /opt/pw-browsers, so Playwright screenshot QA cannot honestly be claimed. Responsive static rules, heading/table/SVG semantic checks, and a zero-warning validator provide fallback validation.