AVUT: When Video QA Can No Longer Guess the Answer from the Question

1. Link Verification and Evidence Boundaries

All links were accessed directly. Page numbers refer to the printed pages of the final EMNLP PDF, with PDF sequence numbers added where needed. The paper was initially titled ACVUBench and renamed AVUT in the final version; these are not two separate works.

“Reported by the paper” refers only to values readable from its PDF text or tables; “recalculated here” uses subtraction and division only. AVUT reports no confidence intervals, standard deviations, significance tests, or annotator agreement, so this report does not claim “significant” superiority.

2. Motivation: A Leaderboard May Measure Question-Reading Skill

AVUT Table 8 (p.6577 / PDF p.9) shows that without video, GPT-4o reaches 68.00%, 65.42%, and 47.57% on VideoVista, TemporalBench, and NExT-QA, respectively, while Gemini reaches 62.63%, 59.37%, and 48.95%—far above the 25% random baseline for four-choice questions. Video-QA accuracy can therefore mix in priors from question wording, answer length, option position, and data sources.

AVUT Table 1 (p.6570 / PDF p.2) contains 2,662 videos, 11,609 QA pairs, and 8 tasks, with an average duration of 67.8 seconds. AV-Human contains 698 videos / 1,734 QA pairs, and AV-Gemini contains 1,964 videos / 9,875 QA pairs. Videos are mainly English-language, collected after 2023, no longer than two minutes, and span 18 audio-centric domains. The six MCQ tasks are Audio Information Extraction (AIE), Audio Counting (ACC), Audio Event Localization (AEL), Audio-Visual Correspondence Matching (AVCM), Audio-Visual Object Matching (AVOM), and Audio-Visual Temporal Matching (AVTM). The open-ended tasks are Audio-Visual Segment Matching (AVSM) and Audio-Visual Diarization (AVDiar). AVDiar contains 260 videos with an average of 3.59 speakers (§3.2, p.6572).

3. Narrative, Assumptions, and Trade-offs

The paper first collects new videos; humans build AV-Human; Gemini 1.5 Pro sees four examples per MCQ task to generate AV-Gemini; all 9,875 generated QA pairs receive human review; GPT-4o then answers each item without video under four cyclic option orderings, and items answered correctly in all four orders are removed. The central assumptions are that stable correctness across permutations indicates a textual shortcut, GPT-4o exposes most such shortcuts, and the gain from full input over a single modality in the same model approximately reflects that modality’s value. The trade-offs are possible overfitting to GPT-4o during filtering, generator style being encoded in the data, and short videos sacrificing long-range understanding.

4. Formalization, Pseudocode, and Pipeline

Let an item be (q=(x_a,x_v,t,O,y)) and the cyclic permutation set be (C_4={ABCD,BCDA,CDAB,DABC}). The filtering indicator is [ S_m(q)=\prod_{\pi\in C_4}\mathbf1[m(t,\pi(O))=\pi(y)]. ] If (S_{GPT4o}(q)=1), the item is removed. Under independent uniform guessing, the ideal probability of four consecutive hits is ((1/4)^4=0.390625%), but model answers are not independent and the protocol covers only (4!/4=6) of all permutations, so this is not a true false-positive rate. MCQ uses micro-averaged accuracy. The same-model modality gain in Table 5 is [ \Delta_{AV-V}=78.34-62.78=15.56\text{ pp},\quad 15.56/62.78=24.78%. ]

New videos ─┬─ Human: 698 videos → 1,734 QA pairs
            └─ Gemini + 4 demos/task: 1,964 videos → 9,875 QA pairs
                              ↓ full human review
Six MCQ tasks → GPT-4o without video × four cyclic options → remove stable text-only hits
AVSM/AVDiar ─────────────────────────────────────────────→ open-ended evaluation
                              ↓
12 models, varied frame/audio budgets → MCQ, matching, DWER, GPT ranking, ROUGE-L

5. Key Mechanisms and Engineering Details

Closed-model evaluation took about 6 hours and open-model evaluation about 4 hours (§4.2, p.6573), but GPU setup, batching, API cost, and retry rate are omitted, so these numbers cannot support an end-to-end efficiency comparison.

6. Dataset, Metrics, and Protocol

MCQ uses accuracy; AVSM reports pair/full matching; AVDiar uses DWER (lower is better); open answers additionally report average GPT-4o rank and ROUGE-L. Frame inputs are not standardized: Video-SALMONN uses 2 fps, VideoLLaMA2 16 frames, PandaGPT 10, Qwen2-VL 1 fps, LLaVA-Video 32, InternVL2 32, VILA1.5 6, VideoLLaVA 8, and SALMONN 16 kHz audio (§4.2). The ranking therefore answers “what can default configurations do?” rather than which architecture is stronger under equal token, latency, or memory budgets.

7. Complete Main-Result Matrix

Each cell is AV-Human/AV-Gemini in percent; bold marks the column best. Source: Table 3 (p.6574 / PDF p.6).

Table 4 (p.6574) overall scores:

Gemini leads the runner-up by 19.96 pp / 34.19% on AV-Human and 24.16 pp / 47.33% on AV-Gemini (recalculated here). The latter cannot be attributed entirely to perception because Gemini generated the data before human revision.

For the open tasks, Table 6 (p.6575) reports AVSM pair/full/DWER of 72.84/26.29/66.31 for Gemini, 38.05/0.00/116.92 for VideoLLaMA2, and 39.87/1.22/155.94 for PandaGPT. Even the strongest model reaches only 26.29 full matching, showing that fine-grained sequence-wide alignment remains difficult. Table 7 (p.6576) reports GPT rank/ROUGE-L/MCQ of 3.331/.108/78.34 for Gemini, 2.233/.074/58.38 for Qwen2-VL, and 2.217/.099/56.52 for LLaVA-Video. Their ordering is broadly consistent, but the paper reports no correlation coefficient, and the highest ROUGE-L is only .108.

8. Ablations, Sensitivity, Phase Transitions, and Statistics

Table 5 (p.6575) gives 78.34, 69.23, 62.78, and 54.10 for the same Gemini model under Audio-Visual, Detailed Transcription, Visual-only, and Audio-only input. AV improves over vision by +15.56 pp / +24.78%, over audio by +24.24 pp / +44.81%, and over detailed transcription by +9.11 pp / +13.16%. Full audio outperforming transcription suggests that prosody, timbre, and synchronization cannot be fully textualized; without a counterfactual interaction term, however, the gain cannot all be proven to come from “reasoning” rather than additional information.

Table 8 (p.6577) reports cyclic text-only GPT-4o/Gemini scores of 41.25/38.75 on VideoVista, 23.22/26.37 on NExT-QA, 16.03/13.05 on Video-MME, and 1.89/4.50 on AVUT. AVUT is 23.11/20.50 pp below random, indicating strong filtering. Yet GPT-4o on TemporalBench rises from 65.42 under ordinary text-only evaluation to 73.57 under cyclic evaluation, so the two columns cannot be read as though cycling must always lower accuracy; the paper also provides no per-item error analysis.

The paper does not report sensitivity to filtering model, permutation count, deletion threshold, four demonstrations, or year cutoff, so it is impossible to determine the purity–scale transition from 4→24 permutations or a 4/4→3/4 threshold. It also does not report random seeds, standard deviations, bootstrap intervals, McNemar tests, video-clustered standard errors, or annotator kappa. Multiple questions from one video are not independent, so 1–2 pp differences between adjacent ranks cannot be declared significant.

9. Technical Lineage of Related Work

Video-MME covers short, medium, and long videos with 900 videos, 254 hours, and 2,700 QA pairs, considering subtitles and audio (abstract and data section). Its remaining gap is that audio is not necessary for every question and text shortcuts are not systematically filtered. AVUT inherits real videos and modality controls, while making audio necessary for its tasks.

AVHBench diagnoses cross-modal hallucination with 2,136 videos, 5,816 QA pairs, and 1,238 descriptions (abstract and §3), but question text can still leak answers. AVUT expands “modality conflict” into “modality necessity + text cannot guess,” carrying forward the idea of separating tasks by error source.

LongVALE has 8,411 long videos, 549 hours, and 105,730 omni-modal events with temporal boundaries (abstract, Table 1). It addresses long-term event perception but is not centered on shortcut-resistant MCQ. Its boundary design flows into AEL/AVTM/AVSM, while AVUT chooses ≤2-minute videos for controllable cost.

MUSIC-AVQA-R diagnoses language and visual bias in MUSIC-AVQA and corrects it through reconstruction and counterfactual training (NeurIPS 2024, §1–3). Its scope is music and tied to a specific training setup. AVUT moves missing-modality stress testing upstream into dataset publication.

General coverage ─ Video-MME ───────────────┐
Cross-modal errors ─ AVHBench ─────────────┤→ AVUT: audio required + new video + text filtering
Temporal events ─ LongVALE ─ boundaries ───┤       ├─ six MCQ tasks
Bias counterfactuals ─ MUSIC-AVQA-R ───────┘       └─ AVSM / AVDiar
                              ↘ not yet unified: long video × all permutations × equal compute

10. Contributions, Reuse, and Transfer

The genuine contributions are threefold: making a text-only model test a release gate; decomposing audio’s role into extraction, counting, localization, object/visual/temporal correspondence, and speaker tasks; and separating AV-Human from AV-Gemini so generator style can be audited. The most reusable idea is “missing-modality control × invariance to answer-option permutation.” It transfers to medical VQA (remove the image), chart QA (remove the chart), document QA (remove the page), RAG (remove the evidence), and robotics (remove the observation). Open answers instead require answer recoverability, semantic equivalence, and counterfactual rewriting; the four-choice rule cannot be applied mechanically.

11. Limitations Stated by the Paper

1. Open-source audio/audiovisual LLMs are relatively old and may be weaker than visual models (Limitations, p.6577), so the leaderboard mixes foundation-model generations.
2. Videos of ≤2 minutes cannot cover long-range temporal reasoning and duration understanding (same page), so results cannot be extrapolated to films, meetings, or lectures.
3. Annotators may prefer visual information, may be affected by their first language, and only four-choice questions are studied systematically (same page), limiting multilingual and open-interaction representativeness.

12. Independent Critique: Because X, Therefore Y

1. [Construction attribution] Because Gemini generated the 9,875 AV-Gemini questions, and Gemini scores 75.21 on that split versus the runner-up’s 51.05, the 24.16 pp gap may include generator-style matching and cannot be attributed entirely to perception.
2. [Filter generalization] Because filtering uses only GPT-4o, while independent Gemini still scores 4.50% on cyclic text-only input versus GPT-4o’s 1.89%, “shortcut-free” means strong suppression only for the tested models.
3. [Permutation coverage] Because only four cycles are tested out of 24 full permutations, non-cyclic swaps and shortcuts based on answer-length combinations may remain.
4. [Fairness] Because inputs range from 6–32 frames and 1–2 fps to unequal audio, while GPT-4o runs on only 2,100 questions, Table 4 is not an equal-compute architectural head-to-head.
5. [Statistics] Because there are no variances, clustered intervals, or significance tests, and multiple questions share each video, adjacent 1–2 pp rankings cannot be treated as stable differences.
6. [Validity] Because the highest open-answer ROUGE-L is only .108 and no MCQ/open-task correlation is reported, MCQ has only coarse-ranking evidence as a proxy for real QA.
7. [Generation quality] Because the appendix categorizes questions needing revision as wrong answer 40%, task mismatch 20%, irrelevant 15%, multiple answers 15%, and other 10%, yet does not report their overall incidence among 9,875 questions, review cost and raw generation reliability cannot be quantified.
8. [Reproducibility] Because copyright policy permits release of YouTube links only, link rot, regional restrictions, and platform transcoding can change the usable sample; long-term reproduction requires URL survival rates and content hashes.

13. Actionable Open Challenges

14. Recent Companion Reading

1. Video-MME v2: a larger-scale video evaluation that can test whether audio remains necessary for every question.
2. VSTAT: emphasizes joint video-spatiotemporal and audio reasoning, making it suitable for testing AVUT’s short-duration conclusions.
3. HAVEN: difficult audiovisual event understanding; its negative samples should be checked especially for missing-modality stress testing.
4. OmniSelect: studies omni-modal information selection and complements AVUT’s question of whether audio is needed.
5. Joint Audio-Visual Reasoning Benchmark: centers joint reasoning and is suitable for cross-evaluation under equal budgets.
6. LongVALE: an important reference for long-video event annotation that directly exposes the extrapolation boundary of AVUT’s ≤2-minute videos.

15. Conclusion

AVUT turns “does the model actually look and listen?” into an executable data-auditing process. Its strongest evidence is the same Gemini model scoring 78.34 with AV input versus 62.78 with vision (+15.56 pp), together with cyclic text-only scores of only 1.89/4.50%. The largest reservations are generator–leader overlap, a single filter, incomplete permutations, unequal compute, and absent statistical testing. It is a benchmark closer to measuring “evidence dependence,” but it is not yet a causally validated instrument for multimodal understanding.