SuperInfer

4.1 Virtual Lag Time

RotaSched uses Virtual Lag Time(VLT) to place three states on one priority axis. From§4.2.2 and its equation(PDF p.6):

]

(S_B) is the TBT SLO and(S_F) the TTFT SLO; t_last is the last-token time, t_arr arrival, and t_run entry into the current running period. Positive VLT means lagging; negative means relatively well served.() controls TBT sensitivity versus TTFT, with(_B,_F) tolerance windows.

4.2 Largest-VLT-First

Each engine iteration:
  If HBM fits every request: fall back to FCFS
  Otherwise:
    1. Compute VLT for waiting/rotary/running requests
    2. Sort into L by descending VLT
    3. Select waiting/rotary requests from L’s head within B_HBM+B_xfer blocks
    4. If space is insufficient, preempt lowest-VLT running requests from the tail
    5. DuplexKV concurrently swaps in/eagerly swaps out; next iteration uses the prepared batch

From§4.2.3 and Figure 10(PDF p.6). B_xfer is the KV-block rotation budget per iteration, converting bandwidth into how much SLO debt can be repaid. The reusable principle is:system capacity should enter scheduling constraints rather than remain hidden in downstream backpressure.

4.3 Avoiding races in DuplexKV

Concurrent H2D restoration and D2H writeback of updated KV can race. DuplexKV eagerly copies full, immutable blocks to DRAM and marks them synced, rather than waiting for preemption. Synced HBM blocks can then be discarded; only unsynced tails require handling. Block-first layouts merge small cross-layer/head segments into roughly 4 MB transfer units, batching same-direction copies into large kernels(§4.3, Figures 11–14, PDF pp.7–8).

7.1 Six workloads × six methods

Figure 16(PDF p.10) covers 3 models ×2 datasets ×6 methodsacross RPS levels, with TTFT/TBT attainment rows. Curves lack point tables/CSV, so exact five-benchmark×six-method values cannot honestly be reconstructed from small plot pixels. The report preserves matrix structure and supported conclusions:

The only main-result magnitude explicitly given in abstract/text is:up to 74.7 percentage points higher TTFT SLO attainment(not relative 74.7%). High-request-rate throughput improves by up to 29.2%and is generally comparable or slightly higher(Abstract,§5.3.6, PDF pp.1,12). Missing means, SDs, CIs, and per-workload peak RPS prevent determining whether 74.7 pp is stable or a single stress-transition maximum.

7.2 Auditable DuplexKV table

Table 1(PDF p.11) transfers bidirectionally for Qwen2.5-32B 16 GB KV(8GB/32768 tokens per direction), with exact values:

Calculation: speedup=1556.15/E2E; distance from Ideal=E2E/41.66. DuplexKV reduces E2E versus Naive by 96.99%and reaches per-direction bandwidth equal to 94.27% / 93.42%of the ideal 192GB/s. The benchmark correctly uses Grace DRAM’s realizable192+192GB/s ceiling rather than advertised900GB/s link capacity.

8.1 Component ablations

Figure 17(PDF p.10) compares vLLM, RotaSched with low budget L(B_xfer=300), RotaSched forced to high budget H(B_xfer=2400), and full SuperInfer. Three conclusions follow:

1. RotaSched(L) improves TTFT attainment, establishing scheduling value;
2. forcing2400 blocks through an inefficient engine harms TBT because compute cannot hide transfers;
3. full SuperInfer supports large budgets while preserving TBT, showing RotaSched/DuplexKV interact multiplicatively rather than as independent minor optimizations.

Figure 17 lacks exact tables. Table 1 decomposes Naive→MS→MS+MK→full duplex, showing the largest gains from bigger segments/merged kernels; eager full duplex further reduces63.14ms to 46.80ms, an absolute 16.34 ms, corresponding to relative gains of 25.88%. Novelty and the largest performance source are not identical.

8.2(): explicit TTFT/TBT Pareto tradeoff

Figure 18(PDF p.10) fixes(_B=_F=0) and sweeps{1,2,3,4,5,10,50,100}. Larger() prioritizes rotary requests sooner, improving TBT but delaying waiting requests and worsening TTFT. Authors suggest() for TTFT-sensitive use and() as balance; larger values give diminishing TBT benefits with substantial TTFT harm. Main experiments choose3.

8.3(_F),(_B): threshold transitions

Figure 19 fixes(,_B=0); when(_F), P99 TTFT rises sharply with little TBT benefit because waiting-request VLT stays suppressed too long. Figure 20 fixes(,_F=0); larger(_B) delays rotary restoration and worsens P99 TBT. Authors recommend(_B<0) for better TBT tails at the expense of TTFT tails(§5.3.2, PDF pp.10–11). These are clear SLO transition regions, not uniformly smooth tuning.

8.4 B_xfer: hardware capacity enters scheduling

Figure 21 sweeps{300,600,1200,2400,4800}, with larger budgets lowering P99 TTFT/TBT(§5.3.4, PDF p.11), though point values are absent. More backlog clears per iteration, increasing DuplexKV dependence. Together with Figure 17, the conclusion is that larger budgets help only when the transfer engine can support them.

10.1 PagedAttention/vLLM: from fragmentation to block KV

PagedAttention adapts OS virtual memory to KV, minimizing waste and enabling sharing. SOSP2023 reports equal-latency throughput gains of 2–4×. Paging improves capacity but scatters request KV into many small segments, impairing CPU–GPU transfers. SuperInfer retains paging while adding block-first physical layouts/merged kernels:preserve logical paging, reorganize physical movement。

10.2 Sarathi-Serve: stall-free batching instead of prefill blocking decode

Chunked prefill/stall-free scheduling reduce interference; OSDI 2024 reports single-A100 Mistral-7B capacity 2.6×, Yi-34B on two A100s of 3.7×、Falcon-180B pipeline parallel 5.6×. It assumes request KV fits GPU memory; chunking cannot admit requests when HBM is full. SuperInfer retains chunking and proactively rotates residency:make iterations composable, then residency rotatable。

10.3 LTR: predicted SJF instead of FCFS

LTR learns relative output-length ordering rather than exact lengths. NeurIPS 2024 reports chatbot latency reductions up to 2.8×and synthetic throughput gains of 6.5×. It needs an offline ranker and cannot overcome exhausted HBM through queue order alone. SuperInfer uses observable waiting/last-token times and can actually evict running requests:replace predicted workload with measured SLO debt。

10.4 NEO: CPU–GPU computation beyond transfers

NEO offloads some attention with KV to CPU, using asymmetric pipelines/load-aware scheduling to expand batches. Abstract T4/A10G/H100 peak gains are 7.5× / 26% / 14%, with stronger CPUs giving A10G up to 79.3%. CPU enters the compute critical path, complicating hardware balance/model support; the baseline cannot run Mixtral-8x7B. SuperInfer keeps computation on Hopper and uses CPU only as a large KV tier:retain heterogeneous pipelines but narrow CPU responsibilities to reduce model coupling。

10.5 Pie: performance-transparent Superchip memory pooling

Pie is closest: predictable accesses/concurrent swapping on GH200 produce reported peak vLLM-relative throughput 1.9×, latency 2×, and equal-performance GPU-memory reduction 1.67×, with FlexGen-relative throughput 9.4×. Pie emphasizes capacity/performance transparency without explicit TTFT/TBT urgency or high-load proactive rotation. SuperInfer turns the CPU pool into an SLO-controlled execution tier, but lacks measured Pie head-to-head results because code was unavailable(§5.2).

                                                  ┌─ Sarathi-Serve: chunked prefill/stall-free batches
PagedAttention: block KV ┤
                         └─ LTR/LightLLM: queue ordering/future-memory prediction
                                  │
NEO: CPU attention ────────────────┼──> SuperInfer: VLT/LVF proactive rotation
                                  │             + DuplexKV full-duplex migration
Pie: GH200 CPU memory pool ────────┘
                                  │
                                  └─ New shift: achievable bandwidth becomes budget B_xfer

Three shifts drive this evolution: operators to KV management, mean throughput to TTFT/TBT SLOs, and discrete PCIe devices to integrated CPU–GPU memory hierarchies. SuperInfer turns interconnect capacity into a schedulable resource rather than backend detail.

12.1 Stated/directly identifiable boundaries

1. Parameters are not automatically predicted from workloads. §5.3.2(PDF p.11) offers qualitative behavior only;(,_B,_F) require application customization, without online-query-distribution prediction.
2. Benefits vanish at low load. Introduction(PDF p.2) states that with ample memory SuperInfer matches baselines. Its value is SLO-aware memory management under pressure, not universal per-token kernel acceleration.
3. Code validation covers one GH200. README’s AE setup uses one GH200(144GB HBM,480GB DRAM); other Ampere/Hopper cards should work but are untested. TP=2 experiments still use related GH200/NVLink hardware.

12.2 Independent critique(evidence→implication)

1. [External validity] Because main experiments use only GH200 NVL2 and a fixed 400GB pool,74.7 pp cannot generalize to PCIe, MI300A, CXL, or heterogeneous cloud instances with different feasible B_xfer ranges.
2. [Baseline fairness] Because Pie is excluded for unavailable code despite being the closest GH200 pooling approach, evidence establishes superiority only over implemented baselines, not the nearest contemporary Superchip system.
3. [Statistical reliability] Because Figures 16–23 lack repeats/error bars/CIs/tests, maximum+74.7 pp may amplify load-transition/runtime noise. Request-level bootstrap and repeated trace-replay intervals are needed.
4. [Threshold dependence] Because main SLOs are TTFT5s/TBT100ms and the quality of(,_B=0,_F=0.5) depends directly on thresholds, balance is not intrinsic. If interactive voice requires TBT<50ms, rankings/transitions may change.
5. [Workload realism] Because Poisson-driven ShareGPT/LMSYS misses bursts, daily cycles, priorities, shared prefixes, and retry correlations, production-burst recovery is unproven. Self-exciting arrivals and real-trace replay are needed.
6. [Contribution attribution] Because merging64KB segments into 4MB alone cuts1556.15→159.87ms(9.73×), while full DuplexKV cuts63.14→46.80ms(1.35×), layout/batching drive the largest gains. They should not all be attributed to duplex transfers or scheduling.
7. [Cost/energy] Because only SLOs, throughput, and latency are reported—not GH200 price, CPU/DRAM power, energy per million tokens, or TCO against two H100s—economic superiority of large-DRAM Superchips with 400GB KV pools is unknown.
8. [Multitenant fairness] Because LVF prioritizes maximum debt without per-tenant slowdown, starvation, or Jain fairness measurements, combinations of priorities, long outputs, and frequent restoration may monopolize service. Tenant weights and maximum consecutive rotations should be explicit.

Theory

• VLT vs. deadline/queueing optimum： Under equal traces, HBM/DRAM, and transfer budgets, compare LVF, EDF, Whittle-index, and estimated SRPT on both SLO violations, worst-case starvation, and competitive ratio. Existing curves do not establish when piecewise VLT approximates optimality.
• Fixed parameters versus online control: On stationary Poisson, Markov-burst, and self-exciting Hawkes arrivals, compare fixed(,), bandit controllers, and MPC, preregistering weighted TTFT/TBT violations plus parameter-switch costs.
• Lossless offloading versus lossy compression: At equal HBM+DRAM byte budgets, compare SuperInfer, KV quantization/eviction, and combinations. Add perplexity, task accuracy, and exact-output agreement to SLOs, forming quality–capacity–tail-latency frontiers.

Engineering applications

• SuperInfer vs. Pie： Run identical ShareGPT/LMSYS traces on the same GH200, vLLM commit, and 400GB pool. Report TTFT/TBT CDFs, throughput, C2C/DRAM bandwidth, CPU use, power, and 30 repeats. This is the key missing comparison.
• GH200 vs. MI300A vs. PCIe H200： Fix models/traces, tune segment sizes/B_xfer separately, and compare fractions of platform-specific physical limits rather than raw GB/s.
• One versus 8 tenants: Assign different SLOs/weights, compare LVF/weighted-LVF, and report per-tenant attainment, P99 slowdown, and starvation to test production QoS.

Emerging directions

• Reasoning models with long CoT: Compare SuperInfer, SkipKV, and combinations on DeepSeek/Qwen reasoning at fixed accuracy/max tokens, testing complementarity or interference between moving full KV and suppressing redundant generation.
• prefill/decode disaggregation： Place rotary tiers in decode pools, remote KV servers, or CXL memory, comparing local GH200. Measure restoration latency under congestion and failure recovery to test cross-node VLT.
• Agents and shared multi-turn context: Compare ContextPilot/prefix caching at equal reuse rates, assessing avoided prefill versus proactive rotation, marginal gains, and cache pollution in long conversations.