Tempo: turning the compressor itself into a small model that reads the question

0. Issue metadata and methodology

1.1 Focal paper

All links were checked through retrieval. ✅ Confirmed accessible; ⚠️ exists but retrieval is restricted or information conflicts; ❌ not found.

1.2 Related work

Small Vision-Language Models are Smart Compressors for Long Video Understanding

Junjie Fei, Jun Chen, Zechun Liu, Yunyang Xiong, Chong Zhou, Wei Wen, Junlin Han, Mingchen Zhuge, Saksham Suri, Qi Qian, Shuming Liu, Lemeng Wu, Raghuraman Krishnamoorthi, Vikas Chandra, Mohamed Elhoseiny, Chenchen Zhu
Meta AI · KAUST — ECCV 2026 — arXiv 2604.08120 (April 2026)

2.1 Positioning in One Sentence

The key is not using a small model for compression—BIMBA and LongVU already compress—but one forward pass with two outputs: relevance is not computed by a separate retriever, but taken from the compressor's own hidden states. Compression and scoring therefore share exactly the same multimodal context, with O(1) routing overhead.

2.2 Motivation and Problem Evidence

The paper frames the problem as a structural mismatch: video streams are dense and variable in duration, while LLM context is fixed and scarce. Its introduction says dense visual streams quickly exhaust token budgets and worsen the lost-in-the-middle problem (citing Liu et al., 2024c).

What is novel about this problem framing? Not that long videos are difficult—an old problem—but that the paper reframes it as a bandwidth-allocation problem rather than compression-ratio problem. Most predecessors optimize a single scalar: tokens per frame (2 for LLaMA-VID, 16 for VideoChat-Flash, 16× compression for BIMBA). Tempo argues that tokens per frame should not itself be a constant.

Why is it worth solving? One quantitative argument comes from the paper's Table 1: Qwen2.5-VL-7B uses 1924 tokens/frame to score 65.1 on Video-MME, while Tempo uses a measured average of approximately 2.8–3.6 tokens/frame to score 67.8. This is about a 687-fold token gap, with a gain of +2.7pp in the opposite direction. It suggests that much of today's visual bandwidth is wasted and that the waste can be systematically recovered.

2.3 The Paper's Argument

The argument proceeds through four observation-to-inference steps:

1. Observation: A video's information density depends strongly on the query. Asking when someone tightens a screw requires a very different temporal distribution from asking for a video summary.
   Inference: Compression must be query-aware. (→ LongVU / BIMBA already reached this point)
2. Observation: Query-aware compression requires a module that understands both vision and text. A small VLM is exactly such a module, already available and pretrained.
   Inference: Instead of designing a new compression operator, repurpose an SVLM as the compressor. (→ First turning point)
3. Observation: While compressing, the SVLM already knows whether the segment relates to the question; it simply has not been asked to say so.
   Inference: Its hidden states can provide at no extra decoding cost a relevance score. (→ Second turning point; the foundation of ATA)
4. Observation: Segment-level scores remove the need to divide the global token budget evenly.
   Inference: Design ATA to distribute the budget by score continuously rather than through binary top-K selection.

Implicit Design Trade-Off(Implicit in the design, though not explicitly stated):

• Sacrifice multiround dialogue efficiency for single-query compression quality—compression is query-conditioned, so changing the question requires recompressing the entire video (acknowledged in Section 6).
• Sacrifice adaptive compression depth for single-pass latency—each segment produces a fixed k_max memory tokens, instead of letting the SVLM autoregressively choose how many to write (acknowledged as “a profound optimization challenge”).
• Sacrifice trainable routing for zero extra cost—relevance scoring is entirely zero-shot, without supervision. This is both an elegant feature and the primary target of critique (see §2.7).

2.4 Methodology and Mechanisms

2.4.1 Architecture overview

2.4.2 Formal definitions

End-to-end objective—note that there is no auxiliary compression loss, routing network, or distillation loss, only a standard autoregressive loss:

Fixed capacity k_max itself acts as a structural bottleneck that induces compression. This is the method's most economical design choice: compression emerges from the training objective rather than explicit supervision.

Zero-shot relevance prior—let h_i^rel denote the SVLM's final hidden state before answering the binary question of whether the segment is relevant to the query, and w_yes / w_no and for frozen language-head weights:

The paper's most elegant engineering step reduces an otherwise autoregressively decoded judgment to one vector inner product. It also emphasizes extracting s_i and H_i in the same forward pass(the forward pass continues after reading the hidden state), ensuring that both outputs condition on exactly the same multimodal context.

ATA's two-stage allocation:

Prefix truncation is justified by the paper's semantic front-loading hypothesis: causal attention packs the most salient evidence into the earliest memory tokens. Ablations strongly support this hypothesis (see §2.5.3-C).

2.4.3 Key mechanisms

2.4.4 Engineering details

• Sampling: Decord, 2 FPS. Training windows contain 4 frames; inference windows contain 8 frames(a train/inference mismatch unexplained in the paper; see critique ⑥ in §2.7).
• resolution: Native image resolution; video long edge capped at 512.
• Hardware: 64-GPU cluster + FSDP. Total training duration and GPU-hours are not provided.
• Evaluation framework: lmms-eval.

Four-stage progressive curriculum(Appendix Table A; module freezing is crucial):

Data mixture(Appendix Table B):

2.5.1 Benchmarks and datasets

This benchmark progression is well designed: duration increases monotonically from 473s to 4101s, directly exposing degradation with video length.

2.5.2 Main-results matrix (Table 1)

pp = percentage point. Bold marks the column best.

Relative-gain calculation(Tempo 4K vs. VideoChat-Flash, the strongest specialized open-source long-video baseline):

Comparison with LLaMA-VID (the most important numbers for the technical lineage)

At almost identical token budgets(LLaMA-VID's fixed 2 tokens/frame versus Tempo's measured mean of 2.8 tokens/frame):

A. Progressive training curriculum

B. Segment-level budget allocation (ATA's central defense)

C. Within-segment token reduction (testing semantic front-loading)

D. Relevance-score sources and zero-shot priors

E. Temporal continuity (minimum token allocation)

k_min = 4 contributes only +0.52pp, the smallest effect among the five ablation groups. It is necessary by design to avoid breaking the timeline, but its quantitative benefit is far below ATA (+2.00pp vs. uniform) and prefix truncation (+2.50pp vs. suffix).

2.5.4 Budget sensitivity: a regime change understated by the paper

2.5.5 Statistical significance

2.5.6 Efficiency analysis: a conspicuous omission

The paper repeatedly emphasizes ATA's O(1) complexity, single forward pass, and avoidance of autoregressive overhead. However,

2.6 Assessment of Value

The Substantive Contribution

Neither small-model video compression (BIMBA already learns compression) nor query awareness (already in LongVU) is new. Tempo's actual contribution is making compression and routing two outputs of the same forward pass.

Previously, query-aware methods faced a tradeoff: heuristic scores (LongVU's DINOv2 similarity plus attention, cheap but semantically indirect) or external retrievers (accurate but expensive, with representations misaligned with the compressor). Tempo offers a third path: the compressor has already considered the question while compressing; simply read out that assessment. Table 2-D provides a clean comparison: the external dense retriever (63.14) is no better than the freely extracted internal prior (63.46).

Independently Reusable Designs

1. Logit-difference projection for zero-shot relevance scoring s = σ((w_yes − w_no)ᵀh). This is general: any causal-LM system needing continuous confidence for a binary judgment can replace autoregressive decoding with an O(1) calculation. Applications include RAG reranking, agent tool selection, and data-cleaning scores.
2. Retain the prefix, not the suffix (semantic front-loading). Table 2-C's +3.88pp (+9.7pp on LVBench) is a reusable empirical finding: causal-attention compressed representations concentrate information at the beginning. KV-cache eviction and token-pruning methods should default to removing the tail.
3. Place memory tokens at the end of the causal sequence. This simple but crucial construction turns compression into attention-based reading without new operators.
4. Continuous allocation > binary top-K as a general principle (Table 2-B, +0.86pp): a modest but clear direction.
5. Train compression and reasoning separately(Stage 3 freezes the entire compression stack and trains only the LLM). The curriculum itself is transferable.

Implications for Future Work

Tempo opens a research space in hierarchical routing. Section 6 proposes turning the global LLM into an active routing agent that requests high-resolution representations of selected segments as needed. This converts one-shot compression and decoding into multiround agentic retrieval. If successful, long-video understanding would shift from compression to agent retrieval, converging with deep research and agentic RAG.

Category 1: author-acknowledged limitations (Section 6, Discussion and Future Works)

1. Routing is entirely zero-shot, without post-training. The authors propose SFT or RL to explicitly develop this capability.
2. Compression depth is fixed and cannot adapt reasoning effort. Inspired by reasoning-model test-time compute, the authors nevertheless acknowledge the substantial optimization challenge of adaptive autoregressive extraction without creating a severe latency bottleneck.
3. Multiround dialogue requires re-extracting all visual features. Query-conditioned compression means a follow-up question requires rerunning the whole video.

Category 2: additional critique (8 independent observations)

① [Underperforming subtask + argument gap] ATA is not optimal on the primary benchmark, LVBench, losing to three simpler alternatives

Hard top-K routing in Table 2-B scores on LVBench 52.7 > ATA's 52.3; Token Merging in Table 2-C scores 53.0 > 52.3; the Tempo SVLM with a standard prompt in Table 2-D scores 53.4 > 52.3. The last is the highest LVBench result in the table, 0.7pp above the main table's claimed SOTA of 52.7. The central narrative makes continuous adaptive allocation crucial for extremely long videos, yet continuous allocation loses to binary allocation on the longest benchmark, contradicting that claim.

A plausible mechanism is that LVBench's 4101s videos yield very large N; ATA's proportional fallback formula k_i = k_min + [B_res · ŝ_i / Σŝ_j] then, because Σŝ_j is too large, compresses each segment toward k_min and becomes approximately uniform. Hard top-K at least gives selected segments full capacity. The paper does not discuss ATA's large-N degeneration, leaving a clear gap.

② [Experimental design] Ablations do not control token budget

In Table 2-A, w/o Stage 3 with w/o ATA are measured at a 16K budget, while the default is at 8K. The claimed Stage 3 contribution of +3.80pp and ATA contribution of +1.34pp therefore both reflect confounding between budget and component. The paper supplies neither 8K-budget w/o ATA controls nor full Tempo at 16K. Table 2-A cannot isolate attribution.

③ [Missing fair comparison] The table structure downplays the gap to the strongest same-family model, Qwen3-VL-8B

Both Tempo components come from the Qwen3 family (Qwen3-VL-2B + Qwen3-LM-4B), but Qwen3-VL-8B leads on all three shared benchmarks in Table 1:

With more parameters but a simpler architecture, this same-family model beats Tempo on every shared benchmark, with gaps likewise growing with duration (−2.5 → −3.6 → −5.3). Tempo is correctly positioned as SOTA at 6B, but the paper does not ask: what happens with Tempo on an 8B backbone, or with Qwen3-VL-8B under the same token budget? Without this head-to-head, the central proposition that compression beats expansion is not established.

④ [Underperforming subtask] LongVU, a 2024 heuristic method, wins on Video-MME Long

LongVU (ICML 2025, 64 tokens/frame, entirely heuristic without learning) scores on Video-MME Long 59.5, above Tempo 4K's 57.8(−1.7pp) and Tempo 8K's 57.0 (−2.5pp). This is the only subtask where an earlier, simpler method overtakes Tempo, and the paper does not discuss it. At a mean 2386s, Video-MME Long lies within Tempo's supposed advantage regime, weakening the claim that adaptive allocation broadly dominates long videos.

⑤ [Argument gap + unexplained counterintuitive result] The default is not the best ablated configuration

As §2.5.3-D notes, the base-model prior with explicit routing prompts averages 63.92 and wins 4 of 5 benchmarks, exceeding the default's 63.46. The paper neither explains why fine-tuning worsens the SVLM as a router nor updates the default. This is a methodological concern: if Tempo training harms discriminative calibration, sharing one forward pass for compression and routing introduces an internal tension—compression and routing objectives may compete. A simple experiment measuring routing AUC at Stage 0/1/2/3 checkpoints could test this hypothesis, but the paper does not conduct it.

⑥ [Experimental design] Training and inference use different segment windows without ablation

Training uses 4 frames/segment, while inference uses 8 frames/segment(§4.1), a 2× distribution shift. The paper neither explains the motivation—presumably covering more frames at fixed B_max —nor analyzes inference-window sensitivity. Because k_max = 128 is unchanged, 8 frames/segment gives each inference frame half its training-time memory capacity. This may be a hidden reason the 4K budget beats 8K: a smaller budget forces more aggressive segment discrimination, partly compensating for the shift. The paper does not connect the two.

⑦ [Reproducibility + metric validity] Efficiency claims lack measurements, while cost grows linearly with duration

As §2.5.6 notes, there are no latency/FLOPs/memory data. More specifically, roughly 1025 forward passes through a 2B model on LVBench may offset global-LLM token savings. The claimed O(1) describes only relevance scoring, not the entire compression pipeline. Presenting this local O(1) claim in the context of an efficiency-focused paper risks misleading readers. Future work should add a three-dimensional latency/memory/accuracy Pareto plot directly comparing VideoChat-Flash and STORM.

⑧ [Statistical validity] No error bars, while most key differences are near the noise scale

At least 5 key Table 2 gaps are ≤0.9pp: ATA vs. top-K 0.86; default vs. base prior 0.46; k_min=4 vs. hard pruning 0.52; 4K vs. 8K 0.22; Token Merging vs. default 0.82. Single runs without seed variance cannot support strong claims that design X beats Y. Just 3 seeds on MLVU and LVBench would substantially improve credibility.

Limitations of My Own Analysis (Disclosure)

• I read the arXiv HTML v1 rendering. The column alignment of Table 1's Actual avg. toks/frame row differed across two retrievals (6 cells versus 7), so I cannot be completely certain which benchmarks correspond to 2.8 / 3.6 / 3.4 / 2.9. This report consistently describes a measured mean of 2.8–3.6 tokens/frame under the 4K budget.
• Only captions were readable for Figure 3 (budget × frame-count sweep) and Figure C (qualitative allocations); exact axis values could not be extracted from HTML. The 12K discussion in §2.5.4 therefore relies only on the caption, without verifying the curves.
• Official ECCV 2026 proceedings were not yet available. The venue comes from self-reporting on GitHub and the project page and could not be cross-checked against an authoritative third party.
• No code was run; all performance figures are author-reported, without independent reproduction.

3.1 LLaMA-VID — November 2023 arXiv / ECCV 2024 main conference (pp. 323–340)

① Problem addressed. Long videos cannot fit into LLM context. LLaMA-VID offered an unusually aggressive answer: just 2 tokens per frame. A Q-Former generates queries from the user's instruction Q_t and aggregates through context attention into one context token; adaptive pooling produces one content token. EVA-G serves as visual encoder. This lets 64K context hold over 3 hours of video. Its core contribution: demonstrating that 2 tokens/frame is technically feasible.

② Unresolved gap. The authors acknowledge this explicitly in their conclusion:

The more fundamental gap is that the allocation of these 2 tokens is static. A crucial action and an empty shot both receive 2 tokens. Table 5 quantifies the cost: compressing content to 1 token loses 2%–6%. The gap is that compression ratio is a global constant rather than a local variable.

③ How the focal work responds. Tempo returns to roughly the same token scale (measured mean 2.8 versus LLaMA-VID's 2) but raises MLVU from 33.2 to 75.6 (+42.4pp) and Video-MME from 25.9 to 67.8 (+41.9pp). Its response is not that two tokens are insufficient, but that assigning exactly two tokens to every frame is wrong. ATA gives some segments 128 tokens and others only 4, while retaining a mean of 2.8.

④ Transfer of key designs. LLaMA-VID's context token(driven by user instructions and aggregating text-relevant visual cues) is the direct ancestor of Tempo's query-conditioned memory token . The transfer is: external Q-Former query generation → placing the query directly in the SVLM causal sequence. The former requires training an additional Q-Former; the latter reuses the SVLM's own attention, making it simpler and stronger.

3.2 LongVA (Long Context Transfer from Language to Vision) — June 2024 arXiv preprint (venue not found)

① Problem addressed. LongVA takes the opposite route: if compression loses information, expand context instead. Its central idea is long-context transfer: continue training Qwen2-7B-Instruct's Text context to 224K (RoPE base raised to 1B; 900M tokens / 1000 steps), then transfer this capability zero-shot to vision without video training. UniRes represents video as one long 336×(336·N) image, with N frames = 144N tokens. V-NIAH remains near perfect at about 2000 frames (≈288K tokens), beyond the training length. The core contribution is demonstrating zero-shot cross-modal transfer of long-context capability.

② Unresolved gap. Three gaps, the first two acknowledged by the authors:

• Nonmonotonic frame scaling: Video-MME scores 52.6 at 128 frames but falls at 384 frames to 51.8(LongVA Table 4). More frames do not imply better performance.
• UniRes has costs: it underperforms AnyRes on low-resolution image benchmarks such as AI2D and ChartQA.
• Exploding cost(my additional observation): the KV cache for 2000-frame inference requires up to 100GB of memory. The 224K limit is imposed by 8×A100-80G memory capacity, not the method itself.

The fundamental gap: expanding context does not decide who deserves bandwidth; it passes the problem to hardware. Uniform 144 tokens/frame has the same allocation structure as LLaMA-VID's 2 tokens/frame, with a different constant.

③ How the focal work responds. Table 1 directly addresses cost-effectiveness: LongVA uses 144 tokens/frame for Video-MME 52.6; Tempo uses about 3.6 for 67.8—40× fewer tokens, 15.2pp higher score. More fundamentally, §2.5.4 finds that Tempo itself degrades when budget doubles (4K 63.68 → 8K 63.46), echoing LongVA's 128→384-frame decline. Tempo's position is that when nominal context has negative marginal returns, the answer is greater information density per token rather than more tokens.

④ Transfer of key designs. LongVA contributes less through an inherited mechanism than by providing a clean counterexample that bounds context expansion. Its V-NIAH protocol—counterfactual questions plus perplexity scoring to prevent language-prior shortcuts—also became a standard diagnostic, continued by VideoChat-Flash's 10000-frame NIAH experiment.

3.3 LongVU — October 2024 arXiv / ICML 2025 main-conference poster

① Problem addressed. First implements query-aware compression as a complete spatiotemporal pipeline. Three steps: (i) DINOv2 computes mean feature similarity in J=8-frame windows and removes frames highly redundant with neighbors, roughly halving frame count; (ii) cross-modal text guidance scores frame features through attention to the query, retaining 144 tokens/frame (12×12) for the top-N_h frames and downsampling the rest to 64 tokens/frame (8×8); (iii) STC spatial token compression prunes spatial tokens in K=8-frame sliding windows when cosine similarity to temporal neighbors is >θ=0.8, reducing tokens further by an average 40.4%. One hour of video finally fits into 8K context, at about 2 tokens/frame on average. The core contribution is establishing the value of query-dependent resolution allocation(Table 3: adaptive 67.62 versus uniform 8k@64 66.84 and 8k@144 66.28).

② Unresolved gap. The lineage's most important gap is that LongVU compression is heuristic and training-free. θ=0.8, J=8, and K=8 are manually chosen; nothing is learned from data. DINOv2 similarity measures visual redundancy rather than usefulness for the current question. Two visually distinct empty shots may both be retained despite being irrelevant to when someone tightens a screw. The authors also acknowledge that video SFT reduces image-understanding ability(Table 10), and GPU limits prevented mixed-data training.

③ How the focal work responds. Tempo replaces the three handcrafted modules with one learnable SVLM: no DINOv2 similarity threshold, no top-N_h hard selection, and no STC cosine threshold. Compression emerges entirely from L_AR the single autoregressive loss. Table 2-B quantifies learned versus handcrafted allocation: ATA exceeds uniform subsampling by 2.00pp and random allocation by 3.70pp. Tempo's four-stage curriculum, mixing images/video/text across ~7.71M samples, also directly addresses LongVU's acknowledged image-capability degradation.

④ Transfer of key designs. Two direct inheritances: (i) different token counts for different segments, from LongVU's two levels (144/64) to Tempo's continuous [4, 128] range; (ii) text-guided relevance scoring, from LongVU's cross-modal attention to Tempo's SVLM yes/no logit difference. The concrete transfer is external attention scoring → internal discriminative confidence”.

3.4 BIMBA — March 2025 arXiv / CVPR 2025 main-conference poster

① Problem addressed. Where LongVU uses heuristics, BIMBA introduces learned compression by inserting a Mamba selective-scan module between the visual encoder and LLM. Frame patches are flattened into a long spatiotemporal sequence; query tokens advance with the scan, and input-dependent gates decide what to write or discard. Compression becomes content-dependent rather than uniform pooling. Three key designs initialize queries through adaptive 3D average pooling of spatiotemporal tokens rather than learned embeddings; queries are interleaved among video tokens rather than appended, removing positional bias; and bidirectional scanning recovers spatial structure lost by 1D scanning. Question conditioning prepends the LLM-tokenized question: Z' = [X; Z; Q]. Compression ratio: >16× (36,864 → 2,304 tokens). The central contribution is demonstrating end-to-end learned compression with subquadratic complexity.

② Unresolved gap. BIMBA has no Limitations section, itself worth noting. Three gaps can be inferred: (i) the compressed token count M and pooling ratios are fixed hyperparameters(1×2×2 / 2×2×2 / 4×2×2 by frame-count range), not adaptive to content or question difficulty; (ii) question conditioning is optional and implemented by prepending rather than architecture, preventing one-pass reuse across questions; (iii) efficiency is shown only through curves, without absolute numbers(no FLOPs, ms, or GB). In short, it learns how to compress, but not how much.

③ How the focal work responds. Tempo addresses (i): ATA makes each segment's k_i vary continuously across [4, 128] with relevance. Uniform subsampling (61.46) in Table 2-B is conceptually a proxy for BIMBA-like fixed-ratio compression; ATA improves on it by +2.00pp. But Tempo does not address gap (iii): as critique ⑦ notes, it also provides qualitative O(1) claims without measurements. This is the only gap left unfilled across two consecutive generations in this lineage.

④ Transfer of key designs. BIMBA's component-ablation methodology is clearly inherited: its Table 1 progresses from learned queries 67.67 to average-pooling initialization 68.61, +normalization 69.85, +bidirectionality 71.68, and +interleaving 73.57. The incremental presentation parallels Tempo's five Table 2 groups. More substantively, it passes on the architectural idea of a learnable compression bottleneck between visual encoder and LLM: BIMBA inserts Mamba layers; Tempo inserts an entire 2B VLM. The transition is specialized sequence operator → general pretrained model, reflecting a broader trend over the last two years.

3.5 VideoChat-Flash — December 2024 arXiv v1 / ICLR 2026 main-conference poster (v4: 2025-07-13)

① Problem addressed. Pushes token efficiency while preserving long-context capability. HiCo (hierarchical visual token compression) has two levels: clip-level video-encoder spatiotemporal attention plus similar-token merging compresses each frame to a mean of 16 tokens; video-level progressive visual dropout during LLM inference removes about 50% in shallow layers and uses attention-guided selection deeper, for total compression of approximately 1/50. It pairs this with a four-stage short-to-long curriculum (4 frames → 8 frames → 64–512 frames → post-tuning at 448 resolution). Context expands to 128K, supporting about 10,000 frames. The core result is that single-hop NIAH retrieval reaches 99.1% at 10,000 frames(versus LongVA's 92% at 3,000 frames and LLaMA-VID's 55%), largely solving long-video retrieval.

② Unresolved gap. Its own data reveal an instructive gap: retrieval is nearly perfect, but reasoning remains far from solved. On multihop NIAVH at 10k frames (Figure 6), CAP is only 31.3% and QA only 25.4%. Finding one frame and reasoning across several are very different capabilities. Mechanistically, HiCo's clip-level compression is query-agnostic: every frame gets 16 tokens without knowing the question. Seemingly minor details crucial in combination for multihop reasoning may be discarded during compression. The authors also acknowledge that video-level compression works only at inference because it is incompatible with training accelerations such as sequence parallelism.

③ How the focal work responds. Tempo directly addresses query-agnostic clip compression by injecting the query into the innermost compression stage. Table 1 shows an advantage growing with duration (LongVideoBench −0.2pp → LVBench +4.5pp): query-aware compression helps most on extremely long videos requiring cross-segment reasoning. Tempo's compression is also shared between training and inference, unlike HiCo's inference-only video-level compression, removing that mismatch.

④ Transfer of key designs. Two inheritances: (i) the short-to-long progressive curriculum is nearly identical: VideoChat-Flash uses 4→8→64–512 frames plus high-resolution post-tuning; Tempo uses 1→8→128→384 frames, with the same four-stage structure; (ii) the demonstration that very few tokens/frame can still achieve SOTA. VideoChat-Flash matches Qwen2.5-VL with 16 versus 1924 tokens/frame, paving the conceptual and empirical way for Tempo to reduce the budget another order of magnitude to mean 2.8. The transfer is a fixed tiny budget → a lower mean budget with dynamic variation”.

4.1 ASCII evolution tree (nonlinear, with parallel branches and convergence)

4.2 The internal logic of the evolution

The driving sequence: Static constants → handcrafted rules → learned operators → pretrained models → adaptive allocation

Three turning points:

1. LongVU (October 2024): from treating all frames equally to query-dependent resolutions. A shift in problem definition that first recognizes bandwidth should be uneven.
2. BIMBA (March 2025): from handcrafted rules to end-to-end learning. A shift in methodology that makes compression policy part of a differentiable objective.
3. Tempo (April 2026): from learning what to compress to also learning how much. A shift in output space that turns compression ratio from a hyperparameter into a model-output dimension.

Shifts in Community Attention:

• 2023–first half of 2024: can it fit? Metrics are maximum frames/context length: LLaMA-VID's 3 hours and LongVA's 2000 frames.
• Second half of 2024–2025: once it fits, can the evidence be found? The metric is NIAH retrieval accuracy: VideoChat-Flash's 99.1% at 10k frames. This was largely solved in 2025.
• Second half of 2025–2026: after finding it, can the model reason? Metrics shift to multihop reasoning and comprehensive understanding of extremely long videos: VideoChat-Flash's reported 25.4% multihop result and LVBench's 4101s. Tempo sits at this transition: its gains concentrate on the longest benchmarks, while the shortest, LongVideoBench, shows a net loss.

6.1 Theory

① What is the upper bound on effective visual-token context?(This report's highest-value, lowest-barrier gap)

This is a concrete measurement problem, not a generic call to study long context. LongVA (0.8pp degradation from 128→384 frames), Tempo (0.22pp degradation from 4K→8K), and STORM (acknowledged shorter effective context) independently observe the same effect, yet none quantitatively characterizes the turning point.

An actionable experiment: Fix the model and video set, sweeping only the injected visual-token count B ∈ {1K, 2K, 4K, 8K, 16K, 32K}. At each B, measure (a) end-to-end accuracy, (b) pure retrieval-probe accuracy such as NIAH, and (c) attention-entropy distribution over visual tokens. The hypothesis: (b) keeps improving while (a) declines beyond some B* , localizing degradation to the Inference rather than Retrieval stage. If confirmed, effective context can be defined as B* = argmax_B accuracy(B), followed by asking whether B* scales with LLM size. This valuable gap could be investigated with one model in a week.

② ATA's degeneration bound at large N

As critique ① infers, when segment count N → B_max / k_min , ATA's proportional fallback approaches uniform allocation. Concrete question: Given B_max, k_min, segment count N , and score-distribution {ŝ_i} entropy H(ŝ), can one analytically bound ATA's performance gap to hard top-K? This would directly explain the LVBench result and yield an adaptive switching rule(automatically switch to top-K when N exceeds a threshold).

③ Do compression and discrimination objectives inherently compete?

Table 2-D shows the fine-tuned SVLM is a worse router than the untuned base model (63.46 vs. 63.92). Concrete experiment: Evaluate routing AUC for Tempo's public Stage0/1/2/3 checkpoints on identical segment/query/relevance-label examples. Monotonically declining AUC would support the hypothesis that compression training harms calibration, pointing directly to a multitask-loss remedy. All four checkpoints are public, making this experiment inexpensive to start.

6.2 Engineering and Applications

④ A three-dimensional latency/memory/accuracy Pareto benchmark

This gap persists across both BIMBA and Tempo. Each provides its own qualitative efficiency claims—curves for BIMBA, O(1) wording for Tempo—preventing direct comparison. Concrete proposal: Fix hardware, such as one A100-80G, and the full LVBench dataset; measure end-to-end wall-clock time, peak memory, and accuracy for LongVU, BIMBA, VideoChat-Flash, STORM, and Tempo. In particular: are Tempo's roughly 1025 2B-model forward passes on a 4101s video actually cheaper than VideoChat-Flash's one-shot hierarchical compression? This is valuable engineering work requiring no new method.

⑤ Compression reuse in multiround dialogue

Tempo's third acknowledged limitation: every new question recompresses the entire video, costing O(rounds × video length). Concrete question: Could a two-level cache combine query-independent coarse compression with query-dependent refinement? Cache the coarse pass once, then refine only top-k candidate segments each round. Key experimental gap: How much accuracy is lost relative to Tempo's single-level approach, and after how many rounds is cost amortized?

⑥ Online / streaming settings

All five generations are offline, assuming simultaneous access to the whole video. Real embodied and video-assistant applications are streaming: the video is still arriving and future questions are unknown. Concrete question: When s_i must be chosen before future segments arrive, ATA's min-max normalization, which depends on global max(s) with min(s))fails completely. An online alternative is needed: sliding-window quantiles, or a learned absolute threshold?This is a clearly defined and currently open direction.

6.3 Emerging Directions

⑦ From one-shot compression to active agent routing

Section 6 proposes a global LLM acting as a router that requests high-resolution segments as needed. Making this concrete: Long-video understanding becomes a tool-calling problem, where the LLM's tool is zoom_in(segment_id, resolution). The missing comparison is: one-shot optimal allocation (Tempo) versus multiround adaptive retrieval (agent), under the same total token budget. This untested head-to-head could determine the subfield's next two years.

⑧ Routing post-training: SFT or RL?

Tempo acknowledges entirely zero-shot routing and proposes SFT or RL, but does not argue which is better—an explicit comparative-experiment gap. SFT requires segment/query/relevance triples, constructible from ground-truth intervals in temporal-localization datasets such as Charades-STA and QVHighlights. RL can use final-answer correctness without intermediate labels, but its signal is sparser. Compare both under matched hardware and data budgets, particularly testing whether RL learns the degenerate policy of always requesting more tokens when reward does not penalize consumption. If so, an explicit budget penalty is needed, reconnecting to effective context in §6.1①.

⑨ Joint multimodal routing with audio and subtitles

Current relevance scores use only vision and the text query. Yet Video-MME with subtitles typically gains several points: VideoChat-Flash 65.3 → 69.7, +4.4pp). Concrete question: Could subtitles/audio provide a cheap routing prior? Retrieve candidate segments through text subtitles first, then run expensive visual compression only on candidates. This could reduce the 1025 SVLM passes in §6.2④ by an order of magnitude. What should the alignment objective be: comparable audio/visual relevance scores on one scale, or independent routing followed by a union? The problem remains insufficiently defined.

Not analyzed in depth here; all are CVPR 2026 Highlights or recent high-profile preprints in Tuesday's vision/multimodality rotation.

A Mixed Diet Makes DINO An Omnivorous Vision Encoder (2026, CVPR 2026 Highlight)

A DeepMind-affiliated team (Kabra, Ovsjanikov, Hudson, Carreira, Mitra, and others) revisits the data recipe for self-supervised visual encoders. Why it matters: DINO is now a default redundancy metric across video-compression methods, including LongVU; improving its generality would propagate through the downstream chain.
CVPR 2026 Highlights

ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention (arXiv 2602.07574)

An efficiency direction orthogonal to Tempo: change how visual tokens enter the LLM—cross-attention instead of concatenation—rather than compressing them. Why it matters: Reading this alongside Tempo separates two independent axes: reducing tokens and changing how tokens are consumed.
arXiv

Video Generation Models are General-Purpose Vision Learners (arXiv 2607.09024)

Argues that video-generation representations transfer directly to discriminative tasks. Why it matters: If true, generative pretraining could replace the entire encoder side of long-video understanding, potentially disrupting this report's full lineage.
arXiv

PersonaVLM: Long-Term Personalized Multimodal LLMs (CVPR 2026 Highlight)

Long-term personalized multimodal LLMs. Why it matters: Directly related to Tempo's need to recompress video for each dialogue turn; personalized memory and query-aware compression offer a natural intersection.
CVPR 2026 Highlights

VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 Oral)

Evaluation work studying VLM strategic capability in multi-agent environments. Why it matters: Reflects the community shift from retrieval to reasoning discussed in §4.2.
Project page

Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning (CVPR 2026)

An RL method grounding multimodal reasoning in perceptual evidence. Why it matters: Directly relevant to routing post-training in §6.3⑧: using RL to learn where to look.
CVF Open Access

Unified Generation and Self-Verification for VLMs via Advantage Decoupled Preference Optimization (CVPR 2026 Highlight)

Unified generation and self-verification. Why it matters: Its advantage-decoupling idea may relate to the competition between compression and discrimination suggested by Table 2-D (see §6.1③).
CVPR 2026 Highlights