DocVXQA: Contextual Explanations, or Answer-Conditioned Support Maps?

0. Link-verification table and evidence rules

All URLs below were actually accessed on 2026-08-29. Experimental numbers come from downloaded ICML/PMLR PDFs read page by page; public code verifies implementation, and web abstracts do not fill unreported values.

Evidence labels: Tables 1–5, Figures 3–4, and Equations (1)–(3) refer to the core PDF; code locations refer to actual GitHub default-branch files. Multiple random seeds, training variance, GPU/time, and energy are unreported; corresponding sections explicitly retain that status.

1. Candidate pool, deduplication, and random selection

Executed by local America/Los_Angeles date date +%u returned 6 (Saturday). First read ml-report-index.md, excluding SAE canonicality, LIMEFLDL, Alignment Potential, and other interpretation/alignment topics covered in the preceding 14 days. The 24-paper pool came from cross-searching ICML/NeurIPS proceedings, OpenReview, arXiv, Hugging Face Papers, Papers with Code, and Semantic Scholar:

Running shuf -i 1-24 -n 1 returned 12. DocVXQA is an ICML 2025 main-conference paper with 21 pages of text/appendices, code, weight links, and descriptions of two datasets; no redraw was required. The full candidate pool is saved separately in candidates_2026-08-29.tsv.

2. Motivation: The answer's location is not its justification

DocVQA takes document image (X) and question (q), producing answer string (Y). Tables, headings, units, column names, footnotes, and layout jointly determine a value's meaning. For “What is the total amount?”, boxing only $180,000 may reproduce the answer without explaining why it is the total or which table/column it belongs to. Pages 1–3 distinguish:

• answer localization: Locate where the answer text appears;
• context-aware explanation: Include the context making the answer semantically valid;
• representation efficiency: Avoid highlighting the whole page.

Existing post-hoc methods face two extremes. Gradient/attention methods are cheap but noisy or simply follow model attention; perturbation methods are closer to causal interventions but require many masked forward passes. Localization supervision applies only to extractive examples whose answers can be copied. The paper operationalizes the tradeoff through three metrics: masked re-answering Accuracy and ANLS measure sufficiency, while retained Pixel Ratio measures minimality(PDF page 6, §4.1).

Direct evidence illustrates the problem: raising raw-attention threshold from 0.05 to 0.50 on DocVQA reduces Pixel Ratio from 0.38 to 0.04, but Accuracy falls from 0.34 to 0.03 and ANLS from 0.46 to 0.08; corresponding PFL-DocVQA Accuracy falls from 0.06 to 0.00 (Table 1, page 6). A single saliency ranking struggles to preserve both the answer and semantic context, beyond simple threshold tuning.

3. Paper narrative, key assumptions, and tradeoffs

The narrative has four steps:

1. Use Pix2Struct to process the full page and generate answers, while deriving a learnable mask (M) from encoder tokens, decoder attention, and positional encodings.
2. Treat (T=X\odot M) as an information-bottleneck representation: masked regions should suffice to predict (Y) while remaining small.
3. Answerability plus sparsity alone degenerates into copying answer text. Use ColPali question/document and answer/document late-interaction heatmaps as token-interaction priors to require broader context.
4. At inference, enclose connected regions in bounding boxes and retain only the top-(k) for readability.

The key assumptions are that a mask is sufficient if the same DocVQA backbone generates the same answer from it, minimal if its area is small, and more context-aware if close to the ColPali prior. These are operational surrogates, not inherently equivalent to fidelity. The tradeoff sacrifices unmasked task performance for explanation learning: on PFL-DocVQA, the method's unmasked Accuracy/ANLS is 0.57/0.79 versus original Pix2Struct's 0.80/0.92 (Table 1), absolute drops of 23/13 percentage points.

4. Equation-level formulation and implementation

Information-bottleneck prototype (PDF Equation 1, pages 3–4):

[ \min_M \mathcal L_{IB}=\beta I(X;X\odot M)-I(X\odot M;Y). ]

The second term represents sufficiency. Since (H(Y)) is constant for fixed labels, autoregressive cross-entropy approximates it:

[ \max I(X\odot M;Y)=H(Y)-H(Y|X\odot M) \Longleftrightarrow \min \mathcal L_{CE}(Y,\hat Y_M). ]

The first term does not actually estimate mutual information as in Equation (1). Kernel-density MI was tried but found sensitive to high-dimensional images and hyperparameters, failing to achieve expected results (pages 4–5, §3.3). It was replaced with L1 sparsity and anisotropic total variation:

[ \mathcal L_{min}=\lambda_1|M|1+\lambda{tv}\sum_{i,j}(|M_{i+1,j}-M_{i,j}|+|M_{i,j+1}-M_{i,j}|). ]

With ColPali prior (M_p), the final objective (Equation 3, page 5) is:

[ \min \mathcal L_{VXQA}=\gamma,\mathrm{MSE}(M_p,M)+\beta\mathcal L_{min}(M)+\mathcal L_{CE}(Y,\hat Y_M). ]

However, paper symbols and public-code names are confusing: PDF Table 4 gives (\gamma=0.5,\beta=5), while code arg_utils.py Moderate gamma=0.5 denotes minimality and beta=5 denotes ColPali interaction; train.py combines them as gamma * loss_mask + beta * loss_mask_colpali, reversing their letter positions in Equation (3). Values do not conflict, but semantic-label conflicts complicate reproduction.

Training:
  full image + question ──> Pix2Struct ──> answer loss (pass 1)
              │ decoder attention / encoder token / position
              └──────────> mask head ──> sigmoid relevance M
  question + image ──> ColPali heatmap ─┐
  GT answer + image ──> ColPali heatmap ├─ add ─> prior Mp
                                        └─ MSE(M,Mp)
  image ⊙ M ──> same backbone ──> answer loss (sufficiency)
  M ──> L1 + TV (minimality)

Inference: question + image ─> M ─> connected components ─> top-k boxes
                         └─ masked image ─> answer

5. Mechanisms and engineering details

Page 4 says the decoder is frozen to preserve text-prediction consistency. However, the public default configuration is decoder_frozen=False, and the training script sets requires_grad=False only when that flag is true; AdamW then receives model.parameters(). Thus the README's direct-training path updates the decoder by default. Unless authors overrode parameters, the paper and public defaults disagree.

6. Datasets, benchmarks, and metrics

• DocVQA: Official pages describe approximately 50K questions and 12K+ document images, with answers as single text spans in images. The paper uses it for conventional document explanations.
• PFL-DocVQA: Invoice domain, originally built from 4,968 manually verified documents and 32,701 newly annotated documents (PFL source paper §4). DocVXQA treats it as a cross-domain/complex-layout test but does not report actual split sizes used.
• Accuracy: Exact/task-defined accuracy; higher is better.
• ANLS: Normalized Levenshtein similarity, tolerant of small OCR/generated-string differences; higher is better.
• Pixel Ratio: Retained pixel area / full image area; lower is more concise, but does not establish causality or ground-truth evidence coverage.
• Human evaluation: 42 participants × 10 images, rating Context and Clarity from 1–5; a second preference experiment uses 12 participants × 21 QA pairs = 252 trials, reporting exact-binomial-style CIs and p-values.

7. Complete main-results matrix

All readable values from Table 1 follow. Each method/threshold reports Accuracy, ANLS, and Pixel Ratio across both datasets; Bold marks the best point estimate per column among explainable methods, but the smallest Pixel Ratio often reflects utility collapse and should not be read as overall best.

Equal-area comparisons are most informative. On DocVQA, DocVXQA .70 and ColPali .50 both have Pixel Ratio 0.23: Accuracy ties at 0.38, while ANLS rises 0.50→0.54, +4 percentage points, or +8% relative. On PFL, area is 4 pp larger (0.22 versus 0.18), while Accuracy rises 0.28→0.43, +15pp / +53.6%; ANLS 0.39→0.66, +27pp / +69.2%. This supports cross-domain value of contextual priors, but is not strictly equal-area.

A closer-area PFL comparison gives DocVXQA .80 at 0.13 pixels versus raw attention .10 at 0.17: despite 4 pp less area, Accuracy rises 0.04→0.33 (+29 pp) and ANLS 0.10→0.54 (+44 pp). This is the main table's strongest evidence.

The upper-bound loss must also be noted: the method's unmasked model trails original Pix2Struct by 5 pp Accuracy/3 pp ANLS on DocVQA and 23 pp/13 pp on PFL. Joint mask training does not preserve original task capability; improved explanations incur substantial backbone-utility costs.

8. Ablations, sensitivity, and transitions

8.1 Three-objective ablation

Table 2 (page 8) row labels are confirmed by the text:

From S to S+M, area decreases by 98%, but Accuracy drops 32 pp and ANLS 29 pp: minimality pushes the model into a collapse regime that merely copies answer characters. Adding TI increases area by 21 pp while recovering 19 pp Accuracy and 18 pp ANLS; the ColPali prior is core anti-degeneration supervision, not a minor adjustment.

8.2 Top-k sensitivity

Table 5 (page 14) lists k=1/2/3/4/5/8/10/15/20/50/100. Accuracy rises monotonically from 0.36 to 0.45, ANLS from 0.52 to 0.60, and Pixel Ratio from 0.21 to 0.40; k=3 gives 0.38/0.54/0.24. The authors call k=3 the best tradeoff, but provide no formal utility function or human study locating a breakpoint. In fact, k=4 adds only 1 pp area over k=3 while gaining 1 pp Accuracy/ANLS. There is no data-established sharp transition, only an author-preferred operating point.

8.3 Weight sensitivity

Only (\gamma=0.5,\beta=5) is reported, without grids, curves, or cross-dataset tuning procedures; code additionally specifies lambda_reg=0.3, alpha_1=2, alpha_2=1, 100 epochs, and seed 42. Robustness to TI/minimality weights cannot be inferred, nor whether area 0.23 is a stable Pareto solution or a single-point choice.

8.4 Statistical significance

The second human evaluation reports 163/252 preferring DocVXQA, namely 64.7%, 95% CI [58.4%,70.6%], p≪0.001(page 9), the paper's only explicit significance result. Main tables, ablations, thresholds, and the first 42-person study provide only point estimates or means±dispersion, without seed/sample-bootstrap CIs. Table 3 does not define ± as SD or SE, preventing precise significance inference.

9. Human evaluation: What is actually demonstrated?

Table 3:

Relative to ColPali, Context improves +0.52 (+13.1%) and Clarity +0.54 (+17.9%); direct preference in the second study is also significant. Thus masks are preferred and more useful for answering from the mask. However, participants see the question and ground-truth answer (page 8), judging support for a known answer rather than answering blindly or diagnosing errors. High scores do not establish better real-world trust calibration.

10. Technical lineage: Problems solved, remaining gaps, responses, and inherited designs

Pix2Struct (ICML 2023)

The paper Uses masked-screenshot→simplified-HTML pretraining and variable-resolution patches to unify vision-language tasks (PDF pages 1–4), providing an OCR-free encoder-decoder for DocVQA. It outputs answers without exposing evidence. DocVXQA retains the 282M base backbone, adds a mask head, and performs a masked second pass. The inheritance is screenshot-parsing representations → learnable patch bottlenecks.

Grad-CAM (ICCV 2017)

The paper Weights convolutional feature maps by target-output gradients to produce localization maps (PDF pages 2–4), without training changes. Autoregressive answers require cross-token aggregation, and gradient maps do not guarantee sufficiency. DocVXQA integrates explanations into training and tests masked re-answering. The inheritance is output-relevance heatmaps → intervenable input masks.

Information Bottleneck attention / DIB-X lineage

IB self-attention Treats masks as compressed representations preserving label information while removing input redundancy; DIB-X further proposes minimal/sufficient/interactive principles. High-dimensional MI estimation remains sensitive; the core paper also acknowledges disappointing KDE-MI results. DocVXQA substitutes L1+TV and answer CE for MI, using ColPali priors for interaction. The inheritance is theoretical mutual-information objectives → three optimizable surrogates.

ColPali (2024)

The paper Combines PaliGemma patch embeddings with ColBERT-style MaxSim late interaction for purely visual document retrieval (PDF pages 3–5). Query-token×image-patch similarities are naturally visualizable, but optimize retrieval rather than answer explanation. DocVXQA uses these maps as teacher priors, merging question-image and answer-image maps. The inheritance is retrieval matching maps → soft supervision for contextual explanations.

CoExVQA (2026, subsequent work)

The paper Chains evidence selection, answer localization, and grounded-region decoding into verifiable explanations (PDF pages 3–6). It addresses DocVXQA masks' conflation of evidence with answer localization, shifting from joint optimization of one heatmap toward staged, verifiable causal paths.

                           ┌─ Grad-CAM / attention rollout: post-hoc correlations
Black-box DocVQA ─ Pix2Struct ─┤
                               └─ IB attention / DIB-X: minimal + sufficient masks
                                         │
ColBERT MaxSim ─ PaliGemma ─ ColPali: query-token × visual-patch interaction
                                         │ teacher prior (Q-image + A-image)
                                         ▼
                      DocVXQA: CE sufficiency + L1/TV minimality + TI context
                                  │
                                  ├─ Single mask + top-k boxes: readable but mixed mechanisms
                                  ▼
                CoExVQA: evidence selection → answer localization → grounded decoding

The field moves from “where is relevance?” to “which regions suffice to support an answer?” and then “can explanation stages be individually verified?” DocVXQA turns retrieval priors into explanation supervision; its risk follows from those priors not being ground-truth reasoning.

11. Actual contributions, reusable designs, and transfer potential

1. Explanations as executable bottlenecks: Heatmaps are inputs that must support a second answer, not mere displays. This explain→intervene→predict structure could transfer to table QA, chart QA, medical reports, and web agents.
2. Report minimality and sufficiency separately: Table 2 clearly exposes collapse from sparsity alone, more informative than a single IoU or attractive visualization.
3. Cross-model teacher priors: ColPali retrieval representations supply context missing from the QA model, a reusable distillation pattern from external semantic priors to internal explanation heads.
4. Human-readable postprocessing: Connected components + top-k boxes turn patch heatmaps into actionable evidence units for review interfaces; k should remain adjustable by users/SLOs.
5. Transfer boundary: Donut qualitative results show interface adaptability, but no quantitative Donut table exists. This is architectural-compatibility proof of concept, not proven model-agnostic performance.

12. Author-stated limitations and independent critique

Limitations stated by the paper

1. MI-estimation failure: §3.3 (pages 4–5) explicitly acknowledges KDE-MI sensitivity to high-dimensional images and hyperparameters and disappointing results. The final objective is IB-inspired, not actual optimization of Equation (1).
2. Generalization scope: The conclusion (page 9) lists more DocVQA architectures/datasets and improved explanation–performance tradeoffs as future work.
3. Backbone-performance loss: §4.3 acknowledges performance loss from jointly handling masked/clear images; PFL's 23 pp Accuracy gap is substantial.

Independent critique (because X, therefore Y)

1. [Supervision leakage / definition] The README and data.py:114–117 explicitly sum question-image and ground-truth answer-image ColPali maps as training targets, so TI learns relevance conditioned on knowing the answer. It is not purely question-conditioned self-explanation and may explain training-answer patterns even when deployed predictions are wrong.
2. [Fidelity circularity] Sufficiency is measured by the same kind of Pix2Struct re-answering masked images, while the mask head reads decoder attention. High ANLS may indicate a model-friendly visual prompt, not faithful causal explanation of the first unmasked inference.
3. [Implementation conflict] PDF page 4 says the decoder is frozen, but code defaults to decoder_frozen=False and optimizes all parameters. The README's default reproduction path conflicts with the paper, leaving Table 1's actual training semantics uncertain.
4. [Baseline fairness] Methods use different thresholds; the ColPali baseline also sees question+answer, while raw attention/rollout/Grad-CAM follow different information/model paths. Rowwise “best” conflates method, information, and area budgets. Equal-Pixel-Ratio Pareto/AUC comparisons are needed instead of selected operating points.
5. [Task-capability cost] PFL unmasked Accuracy falls 0.80→0.57 and ANLS 0.92→0.79, showing explanation training harms core QA generalization. Overall reliability cannot be established only from the explainable-method subtable.
6. [Human-evaluation validity] The 42-person study reveals ground-truth answers and uses only 10 images. It measures apparent support for known answers, not error detection or calibrated trust. “Fosters trust” exceeds direct evidence.
7. [Statistics / reproduction] Main tables lack multiple seeds/CI and the paper omits GPU/time. Public code includes seed42/100 epochs but local absolute paths, preventing assessment of stability and complete reproduction cost. Small 4 pp ANLS gains warrant caution.
8. [Postprocessing attribution] Figure 4 claims that at thresholds ≥0.7 postprocessing improves utility while reducing area, without a complete numerical table; Table 5 shows utility rising monotonically with k. Learned-mask and box-heuristic contributions cannot be isolated.

13. Open challenges: Actionable head-to-head experiments

Theory

• Causal fidelity: Compare DocVXQA, Grad-CAM, ColPali, and CoExVQA at equal Pixel Ratio using comprehensiveness/sufficiency, mask randomization, model-parameter randomization, and counterfactual evidence swaps; test whether heatmap relevance tracks actual changes in original-model decisions.
• Answer-free priors: Train Q-only, A-only, Q+A, and ground-truth-evidence-box teachers, testing fidelity on correct and incorrect predictions; quantify answer-conditioned supervision and misleading-explanation risks.
• Pareto optimality: Jointly sweep β/γ/k, report Accuracy/ANLS–Pixel Ratio curves and hypervolume, choose points on independent validation, and test head-to-head dominance over raw/ColPali curves.

Engineering applications

• Error-diagnosis human study: Hide ground-truth answers; ask participants whether to accept model answers on a half-correct/half-incorrect sample. Compare no explanations, DocVXQA, ColPali, and CoExVQA on human–model accuracy, misplaced trust, and reading time.
• Quantitative cross-backbone evaluation: Use identical evidence metrics for Pix2Struct, Donut, and Qwen-VL/InternVL on DocVQA/PFL/InfographicVQA; report added parameters, two-pass latency, GPU memory, and throughput.
• Reproducible configurations: Run a 2×2 freeze/unfreeze decoder experiment including encoder settings, with ≥5 fixed seeds and mean±SD; identify whether public defaults or paper descriptions produce Table 1.

Emerging directions

• Privacy–explanation conflict: On PFL-DocVQA, test whether precise explanations leak providers/logos/PII; compare DocVXQA with DP-trained backbones, jointly measuring ANLS, membership-inference AUC, and explanation privacy leakage.
• Multipage agent document reasoning: Expand single-page masks into page retrieval→region evidence→answer, comparing token budgets and evidence recall; require page/box traceability for each answer and test cross-page numerical reasoning.
• Verifiable structured explanations: Convert boxes into OCR spans, table cells, or key-value relation graphs; compare pixel heatmaps with structured rationales for human auditing, counterfactual editing, and automated evidence scoring.

14. Recent-work radar (verified links)

1. CoExVQA (2026): Explicitly chains evidence selection, answer localization, and grounded decoding, directly addressing DocVXQA's mixing of inference stages in one heatmap.
2. BBox DocVQA (2025): Its paper card reports 3.6K documents, 32K QA pairs, and bounding-box grounding, supporting evaluation with ground-truth evidence IoU/recall instead of same-model re-answering. Conflicting titles appeared in arXiv aggregate metadata during this visit, so only the Hugging Face paper card is provided as a verifiable entry point.
3. DocMIA (2025): Demonstrates document-level membership-inference risks in DocVQA; explanation maps may create another attack surface and should be jointly privacy-audited.
4. ColPali (2024): Patch-level late interaction remains foundational for visual document retrieval and weak explanation priors, but retrieval relevance is not automatically an answer's justification.
5. PFL-DocVQA / Privacy-Aware DocVQA: Places document QA in federated and differentially private settings; DocVXQA uses its data domain without evaluating explanations' privacy effects.
6. DocVQA 2026 competition: Covers eight document types and emphasizes multimodal reasoning, useful for testing whether DocVXQA works beyond extractive single-page answers.

15. Conclusion

DocVXQA's most credible finding is that under roughly similar mask budgets on PFL-DocVQA, DocVXQA .70 raises Accuracy from ColPali's 0.28 to 0.43 and ANLS from 0.39 to 0.66; direct human preference reaches 64.7% (163/252, p≪0.001). It shows that boxing answer text alone is insufficient and contextual semantic priors substantially improve explanation usability.

The central unresolved issues are equally clear: the ColPali teacher sees ground-truth answers during training, fidelity is evaluated through same-family masked re-answering, public decoder defaults conflict with the paper, and PFL unmasked Accuracy loses 23 pp. The more accurate characterization is: a strong context-support-mask learning method, not an already-validated causal explainer of the original model.

Generation and validation notes: The core paper and ColPali, Pix2Struct, Grad-CAM, and CoExVQA PDFs were downloaded and read; every report link was actually visited, with inaccessible weights/videos explicitly marked. HTML uses the Warm Editorial template, self-contained CSS/JS, dual themes, and responsive tables, and passes structural validation. The environment lacks /usr/bin/chromium or /opt/pw-browsers, so Playwright light/dark/mobile screenshots could not honestly be completed. Responsive CSS, DOM features, and dependency checks were used instead; no browser rendering is falsely claimed. ml-report-index.md has been appended with this entry.