SubSpec: Using the Target Model's Own Low-Bit Shadow as Its Draft Model

Selection and Deduplication Notes

This rotation is Friday: Efficiency and Systems. The pool contained 25 papers from arXiv cs.LG / cs.CL 2510–2608, NeurIPS 2025, ICLR 2026 Oral, ICML 2026, and HF Daily Papers, sampled with shuf -i 1-25 -n 3 . The first draw selected #17 SubSpec; sufficient materials were available, so no redraw.

Deduplication index not updated: this was a scheduled cloud task, and the user's desktop was unreachable; ml-report-index.md was neither read nor written. Deduplication used manual judgment over this session's search results; add this entry later in a desktop session (see the end).

All links were actually attempted in this run on 2026-08-07; outcomes are reported faithfully. ✅ = Successfully fetched and content verified; ⚠️ = URL from trusted search results, but fetching was blocked by the proxy (PROVENANCE_REQUIRED / 403 / bot challenge); content not verified word for word.

2.1 Motivation: Quantifying the Pain Point

The paper begins with a hardware-specific dilemma, not a generic claim that LLM inference is slow: running BF16 Qwen2.5 7B on a consumer GPU with 8GB VRAM, plain parameter offloading decodes at 2.77 tokens/s(Table 1, Qwen2.5 7B / MT-Bench / None); a 32B model under a 24GB limit falls to 0.52 tokens/s. At human reading speeds of roughly 4–5 words/s, 0.52 tok/s means waiting two seconds for each output word—not merely slow, but unusable.

The cause is clear: every target-model forward pass transfers CPU-offloaded weights through PCIe 4.0 x16. The 7B BF16 weights occupy about 15GB; after 11 GPU-resident layers, roughly 10GB must move per token. PCIe 4.0 x16 offers a theoretical ~32GB/s. Thus decoding latency is entirely bandwidth-determined, independent of GPU TFLOPS.

How Existing Solutions Fail

2.2 Narrative and Argument Structure

The argument is not simply “we propose X and experiments show it is better,” but a tight derivation:

Step 1 · Write the Analytic Speedup Formula and Inspect Numerator and Denominator. Section 3.1 gives:

Step 2 · Substitute Offloading Hardware Constants. With offloading, PCIe dominates t_Target, making it 1–2 orders of magnitude larger than GPU-resident t_Draft. Thus 𝒟·t_Draft/t_Target remains O(1) even at depth 48. Conclusion: speedup ≈ τ / O(1). This is the paper's central turning point.

Step 3 · Ask Which Draft Maximizes τ. The limiting case is the target itself as draft. Table 5 measures this ceiling: the Self row gives τ = 34.50 at T=0.2.

Step 4 · Approach That Ceiling Under the “Does Not Fit” Constraint. Since the complete target does not fit in GPU memory, use its 4-bit version—the substitute layers. The rest is engineering: share whatever layers fit, share KV cache, sharpen draft probabilities, and prefetch asynchronously.

2.3 Dissecting the methodology and mechanisms

Key Mechanisms

Configuration Comparison (Implicit in Table 1)

3.1 Hardware and Evaluation Setup

3.2 Main Results Matrix (Table 1, T=0, MT-Bench)

Five-benchmark mean (Table 1, Mean column): Qwen2.5 7B — None 2.68 tok/s, SubSpec 27.09 tok/s (10.10×); Llama3.1 8B — None 2.39 tok/s, EAGLE-2 7.65 tok/s (3.20×).

3.3 Ablations (Table 2; Qwen2.5 7B, MT-Bench, 8GB, Greedy)

3.4 Draft-Temperature Sensitivity (Table 5, τ Values)

This is the most informative table, but the paper does not fully explore it. Three Observations:

1. Sharpening benefits scale with alignment. For EAGLE-2 and the 1.5B draft, reducing temperature from 1.0 to 0.2 is actually slightly harmful(τ decreases by 0.34 / 0.56); Self benefits most (+6.71), with SubSpec in between (+1.79). Sharpening is not universal—pushing a distribution toward argmax only helps when the draft already closely matches the target. Poorly aligned drafts merely become more confidently wrong. The paper does not explicitly identify this mechanism.
2. SubSpec achieves 27.08 / 34.50 = 78.5% of the theoretical upper bound. 4-bit quantization loses 21.5% of acceptance length, a crucial reference for judging more aggressive 2/3-bit quantization.
3. EAGLE-2's τ stays within 3.4–3.8 at both temperatures, barely changing. Acceptance is architecture-limited rather than sampling-limited, explaining why tuning cannot close its offloading gap.

3.5 Two Key Comparisons

Against Direct Quantized Deployment (Table 3; Llama3.1 8B, 8GB)

SubSpec is 5.59× slower than directly running a 4-bit model. The defense is that quantization loses quality, yet the paper reports no downstream task accuracy whatsoever to quantify that loss. Readers therefore cannot answer the real deployment question: is preserving BF16 quality worth a 5.59× throughput cost?

Head-to-Head with SpecExec (Table 4; Llama3.1 8B, 8GB Budget)

SubSpec's throughput is 1.88× SpecExec's, without an extra draft model. The technical criticism is specific: pruning and early exit introduce dynamic shapes that obstruct torch.compile—a judgment grounded in engineering practice rather than abstract complexity.

3.6 Reasoning-Model Results (Table 6, Long Chain of Thought)

Stochastic-Decoding Degradation: At T=0.6, throughput falls roughly 60%, with speedup dropping to 5.8×–7.8×. The deep tree (𝒟=48) strongly depends on low-entropy output, a point insufficiently discussed.

Statistical significance: The paper reports a standard deviation across 5 independent MT-Bench runs of 0.101 tokens/s(relative standard deviation 0.40%). This covers only one configuration on one benchmark; roughly 60 other Table 1 values are single runs without variance.

3.7 Key Hyperparameters

Basis for Choosing 𝒟 = 48(Figure 6): the paper says the optimum occurs where speculation and verification times are approximately equal, but provides only a qualitative plot, not a numerical depth-sweep table. Notably, 𝒟=48 with τ=27.08 implies about 56.4% per-token acceptance: the back half of the draft tree, layers 28–48, is wasted on average, leaving clear room for adaptive depth.

4.1 What Is the Real Contribution?

Not the mere use of a quantized draft—ML-SpecQD(2025-03) already directly cast MXFP4 drafts. SubSpec contributes at three levels:

1. Redefining the Offloading Optimization Objective. Substituting PCIe-dominated hardware constants into the speedup formula yields the conclusion that draft time is nearly free and τ should be maximized. The draft design space flips from small and fast to large and accurate.
2. Maximizing Sharing as an Alignment Mechanism. Weight and KV-cache sharing do more than save memory: they provide a constructive guarantee of alignment. Draft and target see exactly the same attention context; only quantization introduces error. vLLM proposal #39427 's independent preliminary experiments quantify the effect: KL divergence between substitute draft and target is 0.1176, versus 0.5899 for a conventional small draft—a 5.02× difference.
3. A Truly Plug-and-Play Route. No training, no same-family small model, no target-weight changes. Any newly released dense model can use it immediately, without waiting for a draft ecosystem.

Independently Reusable Designs

5.1 Author-Reported Limitations (Appendix A and Conclusion)

1. Minimum VRAM threshold: Qwen2.5 7B needs about 7.1 GB minimum GPU memory. An 8GB card retains only 0.9GB of headroom, leaving almost no flexibility.
2. Quantization granularity unexplored: only 4-bit substitutes are studied; 2-bit / 3-bit are untested.
3. Architecture scope: SubSpec primarily applies to dense LLM architectures; MoE is future work.

5.2 Additional Critique (Eight Points)

Critique 1 — “Lossless” Is the Biggest Claim, Yet No Number Supports It Argument gap / metric validity

The title, abstract, and conclusion repeatedly emphasize losslessness, but the paper contains no downstream task accuracies: no GSM8K accuracy, HumanEval pass@1, output-match rate, or distribution test. Losslessness rests entirely on the architecture-level argument that speculative verification theoretically preserves distributions. Yet draft sharpening uses T=0.2, and stochastic T=0.6 results are reported without formalizing whether correction sampling handles sharpening correctly. A half-page experiment—tokenwise agreement with pure-target output under fixed seeds—would close the gap, but is absent.

Critique 2 — Missing the Direct Predecessor ML-SpecQD Undermines Novelty Claims Missing comparison with a key competitor

ML-SpecQD(Georganas et al., 2025-03-17) already proposed directly casting MXFP4 drafts from BF16 targets, plug-and-play without pretraining or alignment, strongly overlapping SubSpec's construction six months earlier. This report checked the complete bibliography: neither ML-SpecQD nor any work centered on quantized drafts is cited. The closest citations, CLASP [8] and SWIFT [40], skip layers rather than quantize. Presenting training-free low-bit substitutes as unprecedented is therefore unsupported. The real distinction is offloading-specific optimization and τ reaching 27, which requires head-to-head evidence, not omitted citations.

Critique 3 — The Quantized-Baseline Comparison Structurally Avoids Quality Experimental design issues

Table 3 gives standalone HQQ int4 135.84 tok/s versus SubSpec 24.29 tok/s—5.59× slower. The defense is that quantization inevitably harms quality, but without accuracy reporting (critique 1), neither side has evidence. More sharply, SubSpec constructs its substitutes using HQQ 4-bit, the same method reaching 135.84 tok/s in Table 3. If HQQ 4-bit quality loss is practically negligible, as commonly experienced for 7B+ models, SubSpec's entire value proposition must be re-established. This is the most necessary and most conspicuously missing experiment.

Critique 4 — The Crucial 𝒟=48 Hyperparameter Has Only Qualitative Support Insufficient sensitivity evidence

Depth is the largest configuration difference from predecessors: EAGLE-2 uses 6, the small draft 32, SubSpec 48. Figure 6 says speculation and verification times become similar at 48, but no table reports τ and tok/s for 𝒟 ∈ {8,16,24,32,48,64,96}. Thus (a) plateaus or transitions in the throughput–depth curve cannot be identified; (b) optimality across model sizes is unknown—32B has much larger target time and should theoretically favor deeper drafts, yet all use 48; (c) per-token acceptance 27.08/48 = 56.4% means the final 20 levels usually do useless work, with adaptive-depth potential unquantified.

Critique 5 — Statistical Coverage Is About 1.6%Reproducibility risks

Only one MT-Bench configuration reports five-run standard deviation, 0.101 tok/s. Among roughly 60 cells in Table 1's 4 models × 5 benchmarks × 3–4 methods, only one has variance information. This is particularly risky under offloading, where throughput depends on PCIe contention, CPU page-cache hits, and DDR5 channel state. Existing data cannot separate real task differences from noise in the 9.6×–15.3× cross-benchmark range.

Critique 6 — The Speedup Denominator Amplifies the Result Without Qualification in the Abstract Metric validity

“12.5× speedup” uses 0.52 tok/s as denominator: plain BF16 offloading, a configuration consumer users would almost never actually choose, since they would run 4-bit quantization. Under a strict lossless requirement, the comparison is fair, but the abstract gives no qualifier. Relative to direct 4-bit deployment, SubSpec is 0.18×(a slowdown). A systems paper's headline should explicitly disclose its baseline's realism.

Critique 7 — Architecture Coverage Diverges from Real 2026 Local-Deployment Trends Scope of applicability

The paper limits applicability to dense models, while local deployment rapidly shifts toward MoE (gpt-oss, Qwen3-MoE, DeepSeek-V2-Lite), whose large-total/small-active parameter structure naturally suits offloading. MoE's bottleneck is loading experts on demand, so substitutes do not transfer directly. Contemporary DraftExpert(2026-07), designed specifically for edge MoE self-speculation, improves throughput only 1.45×(draft acceptance 84–87%, prefetch hits 86–88%). The 1.45× versus 12.5× contrast suggests SubSpec's huge speedup largely reflects the particular pathological case of dense offloading rather than proof of general applicability.

Critique 8 — Long Context Is Unevaluated Despite Offering the Largest τ Gains Experimental design issues

All experiments use a static KV cache of length 2048 and a stated minimum 7.1 GB VRAM. Together, these leave almost no room for longer cache at 8GB. Yet Table 6 establishes τ as high as 41.68 for long chain of thought, the method's strongest setting, which naturally requires long context. No VRAM–throughput curves at 4K / 8K / 16K are given. The method's best application lies outside the evaluated setup.

6.1 FlexGen — 2023.03, ICML 2023 Oral

Problem addressed. First systematically answered how one commodity GPU can run a 175B model. Linear programming optimizes placement across GPU / CPU / disk, with 4-bit compression enabling OPT-175B on one 16GB GPU at 1 token/s(effective batch 144). In one sentence, it turns “does not fit” from a binary question into scheduling optimization.

What gap remained? All gains rely on large batches, optimizing throughput rather than latency. Effective batch 144 creates unacceptable single-user waits. SubSpec explicitly calls FlexGen throughput-oriented and unsuitable for latency-sensitive online inference. The gap: no acceleration for interactive single-request (batch=1) offloading.

The Core Paper's Response. SubSpec uses batch=1 throughout, directly targeting what FlexGen avoids. It creates parallelism not across requests but inside a single request: one target pass verifies 27 tokens, collapsing 27 serial decoding steps into one. FlexGen exploits inter-request parallelism; SubSpec exploits within-sequence parallelism. They are orthogonal.

Transfer of key designs. FlexGen's 4-bit weight compression to reduce transferred bytes is repurposed: rather than compress the target and lose quality, SubSpec constructs a GPU-resident draft. The same technique shifts from reducing transfer volume to reducing transfer count.

6.2 Speculative Decoding (Leviathan et al.)— 2023.05, ICML 2023

Problem addressed. Established draft-and-verify: a cheap approximate M_q autoregressively drafts γ tokens, then M_p verifies them in one parallel pass, using correction sampling to make the output distribution exactly identical. In one sentence, to direct target sampling while dividing sequential decoding steps by acceptance length.

What gap remained? The framework requires a small model well aligned with the target, practically demanding a same-family checkpoint or extra training. SubSpec notes that existing approaches often rely on pretrained same-family weights, still need training for custom targets, and training-based drafts yield only moderate speedups. The gap: obtaining the draft remains an unsolved supply problem, with alignment its bottleneck.

The Core Paper's Response. SubSpec transforms draft acquisition from a search problem into a construction problem. The target supplies the material, and HQQ—data-independent, taking minutes for 7B–70B—is the construction operator. The supply problem disappears.

Transfer of key designs. SubSpec §3.1 directly inherits and adapts Leviathan's speedup expression. Its insight is essentially substituting extreme hardware constants into that formula and reading the consequence—a good example of reusing a theoretical tool correctly.

6.3 SpecInfer — 2023.05, ASPLOS 2024

Problem addressed. Upgrades linear draft sequences to token trees, verifying all candidate paths in one target pass. It achieves 1.5–2.8× distributed-inference speedup and 2.6–3.5× offloading speedup, increasing stochastic verification success from 52–57% to 96–97%. In one sentence, Tree structure raises the expected return of one verification from one path's length to the best length among multiple paths.

What gap remained? The tree is still generated by an external small model, with width constrained by draft quality. It establishes more candidates yielding greater acceptance but does not locate the candidate-quality ceiling. The gap: tree structure solves verification parallelism, not draft alignment.

The Core Paper's Response. SubSpec inherits tree verification but makes trees deep and narrow(k=6, 𝒟=48), rather than SpecInfer's wide and shallow structure. Width hedges against inaccurate drafts; depth extracts long-range gains from accurate drafts.

Transfer of key designs. Token-tree verification is fully retained, but tree shape flips from width-dominated to depth-dominated—optimal configuration shifts for the same mechanism under different alignment regimes.

6.4 EAGLE-2 — 2024.06, EMNLP 2024 Main (pp. 7421–7432)

Problem addressed. Finding acceptance depends on context as well as position motivates context-aware dynamic draft trees. Across three model families and six tasks, up to 5× speedup, 1.3× EAGLE-1, while preserving distributions. In one sentence, adapt tree shape to context instead of fixing a template.

What gap remained? The draft is a trained single-layer feature-level autoregressive head, minimizing draft time—a sound direction in datacenters. But acceptance has a low ceiling: SubSpec Table 1 measures τ=3.73–3.90, and Table 5 barely changes across temperatures 0.2 and 1.0 (3.42 vs. 3.76), indicating an architectural limit, not a tuning problem. The gap: when target time is enormous, optimizing for a lightweight draft is itself the wrong direction.

The Core Paper's Response. This is SubSpec's clearest target and strongest argument. Table 1 gives EAGLE-2 only 2.89× at 8GB offloading, losing to a plain 1.5B draft at 5.70× and trailing SubSpec's 9.15× by a factor of 3.17.

Transfer of key designs. SubSpec deliberately abandons EAGLE-2's dynamic tree for a static k=6, 𝒟=48 tree, because dynamic shapes obstruct torch.compile. It inherits the problem framing while rejecting the technical solution.

6.5 SpecExec — 2024.06, NeurIPS 2024

Problem addressed. The first work designing speculative decoding specifically for consumer offloading. Modern LLM token distributions are sharply peaked; this motivates large candidate-cache trees producing up to 20 tokens per target iteration. For 50B+ models: 4–6 tok/s with 4-bit and 2–3 tok/s with 16-bit weights. In one sentence, it turns sharply peaked probabilities into an argument for large-scale parallel speculation.

What gap remained? It still requires an independent external draft whose memory competes with resident target layers. SubSpec Table 4 quantifies the cost: SpecExec (budget=512) peaks at 8.79 GB, over the 8GB budget, reaching only 12.95 tok/s and τ=12.93. The gap: large-scale speculation is the right idea, but the draft's memory and alignment bottlenecks remain.

The Core Paper's Response. SubSpec retains deep / large-scale speculation (𝒟=48) and resolves the bottleneck through (a) the target's quantized shadow, giving controlled memory and high alignment, τ 12.93→28.35; and (b) fixed shapes, enabling torch.compile(max-autotune) . Result: 24.29 versus 12.95 tok/s, a 1.88× advantage within budget.

Transfer of key designs. It inherits “peaked distributions enable deep trees,” then uses probability sharpening to actively create that peakedness—from exploiting an existing property to reinforcing it.

6.6 ML-SpecQD — 2025.03, arXiv (Not cited by the core paper)

Problem addressed. Proposes obtaining drafts by directly casting MXFP4 weights from the BF16 target, without separate pretraining or alignment, and recursively accelerating draft generation through multilevel speculation. Single-level gains reach 2× and multilevel gains reach 2.72×. In one sentence, A draft can be a purely numerical projection of the target, rather than another model.

What gap remained? Its 2–2.72× gains are much smaller than SubSpec's, likely because it does not optimize specifically for offloading. In normal GPU-resident settings, target time is too small to amortize the quantized draft's high cost. It establishes the construction, but not the hardware regime where it is most valuable.

The Core Paper's Response. SubSpec adds layer sharing, KV sharing, deep trees, sharpening, and asynchronous prefetch to the same construction in a PCIe-dominated setting, turning 2.72× into 12.5×. Because ML-SpecQD is not cited, this response is reconstructed by this report, not the paper's own positioning. The true contribution is finding the right regime for quantized drafts and engineering it deeply, not inventing quantized drafts.

7.1 Internal Logic: Three Changes in Draft-Model Origin

The driving force is not increasingly clever algorithms, but shifting hardware bottlenecks forcing a rewritten objective. If GPU compute is the bottleneck, drafts must be cheap; if PCIe bandwidth dominates, drafts may be expensive as long as they are accurate.

Key Transitions:

• Transition 1 (2023.05): from optimizing one forward pass to reducing the number of passes, moving acceleration from operators to decoding schedules.
• Transition 2 (2023.05 → 2024.06): from one draft sequence to a tree, expanding the design space from one dimension (γ) to two (k, 𝒟).
• Transition 3 (2025.09): from another model to another precision of the same model. The draft no longer needs alignment; it is aligned by construction, with error only from quantization. KL falling from 0.5899 to 0.1176 quantitatively marks this shift.

Shifts in Community Attention: Early work (2023) emphasized speedup itself; the middle period (2024), losslessness and distribution fidelity; current work (2025–2026), deployability under a fixed hardware budget. SubSpec consistently imposes 8/12/24GB programmatic limits, adopting a consumer perspective rather than asking how much faster an A100 is. That shift itself marks maturation in efficiency research.

The paper supplies a speedup formula but does not use it to interrogate its own results. Here Table 1 measurements are used to recover the implicit denominator 𝒟·t_draft/t_target + γ:

Source: Table 1, Qwen2.5 7B / MT-Bench / 8GB, T=0; denominator = τ ÷ speedup.

1. SubSpec's τ is 7.26× EAGLE-2's, but its speedup is only 3.17×. Draft overhead absorbs the difference: the denominator grows from 1.291 to 2.960 (2.29×). The full 4-bit draft is therefore not free; §3.1's qualitative claim that draft time is negligible under offloading does not fully hold in the paper's own data.
2. The conversion rate per unit τ decreases monotonically(0.775 → 0.465 → 0.338). Heavier drafts make each extra accepted token costlier—a clear diminishing-returns curve the paper does not plot.
3. Quantifiable Performance Headroom: If SubSpec reduced draft overhead to the 1.5B draft's denominator, 2.153, while retaining τ=27.08, speedup would reach 12.58×—37% above today's 9.15×. This points to a concrete next step: speculate again inside the substitute layers (multilevel speculation), or structurally prune them. The gain ceiling can be calculated in advance rather than discovered by luck.

10.1 Theory

(1) Under VRAM Budget B, What Is the Jointly Optimal Projection Operator and Tree Shape? Formalize constrained optimization: under mem(substitute) + mem(KV) ≤ B maximize τ(bits, shared_layers) / (𝒟·t_draft(bits)/t_target + γ). SubSpec samples only one point in this space. Concrete Experimental Gap: Fix 8GB and sweep bits ∈ {2,3,4,8} × shared_layers ∈ {0,4,8,11} × 𝒟 ∈ {16,32,48,64,96}, reporting full τ and tok/s surfaces. High value, low barrier: 90% of the required code already exists in SubSpec.

(2) The Transfer Function from Quantization Error to Acceptance Length. Table 5 provides two anchors: unquantized Self τ=34.50 and 4-bit τ=27.08, a 21.5% loss, with no intermediate points. Concrete question: Is τ(bits) smooth or does it have a transition? A 3-bit cliff makes 4-bit the only viable point; if 3-bit retains τ≈24, using the saved 2.5GB for longer KV cache may win overall.

(3) Optimal Stopping for Adaptive Draft Depth. Measured per-token acceptance is 56.4%; the tree's tail mostly does useless work. Concrete question: Given the draft's confidence trajectory, what is the optimal stopping rule? A solution should reduce expected depth below 30 without losing τ, directly realizing the previous section's calculated 37% headroom.

10.2 Engineering and Applications

(4) No Empirical Losslessness Verification Protocol. The speculative-decoding literature broadly claims losslessness but rarely specifies verification protocols. Concrete Recommendation: standardize fixed-seed reporting of (a) exact tokenwise agreement, (b) estimated output-distribution KL at T>0, and (c) confidence intervals for downstream accuracy differences. This should become a submission requirement.

(5) Missing Long-Context Offloading-Speculation Benchmarks. Existing work almost always uses 2K context. Specific gap: Build benchmarks at 8GB/24GB with ctx ∈ {2K, 8K, 32K, 128K}, reporting VRAM allocation between substitutes and KV, τ versus context, and second-order effects if KV itself must be offloaded. Table 6 suggests higher τ for long outputs, but long outputs are not long inputs; they remain unevaluated separately.

(6) The Form of Substitute Layers for MoE Offloading Is Unsettled.Concrete question: Should an MoE substitute use (a) low-bit versions of every expert, (b) full-precision frequent experts and low-bit long-tail experts, or (c) a distilled dense substitute? Their VRAM / τ / prefetch-hit tradeoffs are unknown. Suggested Experiment: Compare the three substitute constructions head-to-head on DeepSeek-V2-Lite / Qwen3-MoE with the same 8GB budget and PCIe 4.0 hardware.

(7) SubSpec vs. ML-SpecQD Is the Most Urgent Head-to-Head Gap. Their constructions closely resemble one another, yet speedups differ 4.6× (2.72× vs. 12.5×). How much comes from offloading versus residency, the four engineering optimizations, or HQQ versus MXFP4 quantization quality?Recommendation: Compare ML-SpecQD / SubSpec / SubSpec-minus-optimizations on identical hardware, VRAM budget, and target model. Until then, SubSpec's novelty is only partially established.

(8) Quality–Speed Pareto Frontier for SubSpec vs. Direct 4-bit Deployment. Worth a short paper: report (accuracy, tok/s) for BF16, HQQ-int4, AWQ-int4, and SubSpec on GSM8K / HumanEval / MMLU / MT-Bench. If HQQ-int4 loses under 1 pp while running 5.59× faster, SubSpec narrows to domains allowing absolutely no quality compromise. This experiment directly determines the market size of this technical route.

10.3 Emerging Directions

(9) Multilevel Substitute Speculation. A 2-bit substitute drafts for a 4-bit substitute, which drafts for the BF16 target. ML-SpecQD already demonstrates multilevel feasibility (2× → 2.72×, +36%). Concrete Target: Reduce the denominator from 2.960 below 2.2 to realize the theoretical 12.58× ceiling.

(10) On-Demand Precision for Substitute Layers. Layers vary greatly in quantization sensitivity. Concrete question: With fixed total VRAM, how much can τ improve by assigning 2/4/8-bit precision by sensitivity? This mixed-precision search is highly complementary to SubSpec.

(11) Cross-Request Substitute Sharing and Multitenancy. All SubSpec experiments use batch=1. In small multitenant settings, such as home servers supporting 2–8 users, all requests can share substitutes while batching amortizes target-layer transfers. Concrete question: At what batch size does SubSpec's advantage disappear? That crossing defines its boundary with FlexGen-style systems.

(12) Energy Consumption. All related work uses tok/s as the sole metric, but for consumer / edge deployment, J/token is equally important. PCIe transfers consume far more energy than GPU computation; reducing transfer count 27× may yield even greater energy than throughput benefits. Specific gap: No offloading-speculation paper currently reports energy data.

Kimi Linear: An Expressive, Efficient Attention Architecture (2025.10, arXiv)
A production-grounded hybrid of linear and full attention. Worth reading for engineering criteria governing the mixture, not merely academic ablations.
arXiv 2510.26692

MiniMax Sparse Attention (2026.06, arXiv)
Another production validation of sparse attention, contrasting with Kimi Linear's linear route. Read alongside this issue's core paper: both address efficiency, one through architecture and one through decoding schedules.
arXiv 2606.13392 · HF Papers

GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference (2026.07, arXiv)
Chooses attention type layer by layer rather than fixing a global ratio. Methodologically parallels SubSpec's per-layer BF16 versus 4-bit substitution: heterogeneous resource allocation at layer granularity.
arXiv 2607.24788

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling (2026.07, arXiv)
“Done Right” suggests systematic correction of earlier hierarchical sparse attention, a kind of paper often containing valuable failure analysis.
arXiv 2607.02980

EAGLE-3: Scaling up Inference Acceleration via Training-Time Test (2025.03, NeurIPS 2025)
The newest EAGLE generation. SubSpec compares only EAGLE-2; EAGLE-3 under offloading remains open and is a useful additional baseline for reproduction.
arXiv 2503.01840 · OpenReview · Code

KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction (NeurIPS 2025 Oral)
Query-independent KV compression with 3–4× memory reduction and 2× lower latency. Highly complementary: KV cache, limited to static 2048, is a SubSpec bottleneck. Combining KVzip may directly resolve critique 8's long-context problem.
NeurIPS Poster · Code

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference (2026.07, arXiv)
Direct successor addressing the MoE gap. Its 1.45× versus SubSpec's 12.5× is the clearest reference for how unusual dense-offloading pathology is.
arXiv 2607.24434

TileLang: Bridge Programmability and Performance in Modern Neural Kernels (ICLR 2026 Oral)
A kernel DSL achieving up to 5× over Triton on H100. SubSpec heavily depends on torch.compile(max-autotune) and explicitly criticizes dynamic shapes for obstructing compilation; kernel toolchain improvements directly change these systems' design constraints.
ICLR 2026 Oral

1. Unavailable primary material: OpenReview reviews and rebuttal were blocked by a bot challenge; the GitHub README was blocked by proxy restrictions. This report therefore cannot be verified what reviewers questioned or how authors responded, and cannot verify practical reproducibility. All Section V critiques derive from the paper and may overlap with points already discussed in review.
2. How table data were obtained: all SubSpec numbers were structurally extracted from arXiv PDF v2. Apparent discrepancies, such as EAGLE-2 speedups of 2.89× / 2.91× / 3.20×, were checked and traced to MT-Bench versus five-benchmark means versus different targets. Final values identify tables and configurations, but isolated extraction errors remain possible; check the original before citing.
3. Figure 6's draft-depth sweep values could not be read: only its textual description is available, so claims in 10.1(3) are inferred rather than measured.
4. Index not updated: this was a scheduled cloud task; the desktop was unreachable, and ml-report-index.md was neither read nor written. Suggested manual entry:
   2026-08-07 | Efficiency and Systems | SubSpec: Speculate Deep and Accurate | NeurIPS 2025 | Under offloading, draft alignment should be maximized rather than speed; the target's own 4-bit substitute raises τ from 3.7 to 27–42, giving 9.15×–13.76× speedup
   Related work covered (do not repeat within 14 days): FlexGen, Leviathan SD, SpecInfer, EAGLE-2/3, SpecExec, ML-SpecQD, Dovetail, DraftExpert, KVzip.