QServe: Low Bit Width Does Not Guarantee High Throughput

0. Verified links and evidence conventions

All links below were accessed on 2026-08-28. Numbers come only from downloaded paper PDFs read page by page. Web pages verify identity, versions, and resource availability; secondary pages do not supply experimental figures.

Source-label conventions: “Tables 2/4/5” and “Figures 16/17/19” refer to QServe v3 PDF. “Recalculated in this report” means only division, subtraction, or geometric interpretation, never guessing unlabeled exact values from plots. The paper reports no repeated trials, standard deviations, confidence intervals, or significance tests; none are fabricated here.

1. Candidate pool, deduplication, and random selection

First reviewed ml-report-index.md, excluding DuoAttention, PARQ, Q-VDiT, and near-duplicate topics covered in depth in the preceding 14 days. Cross-searching ICML/MLSys 2025 proceedings, arXiv, OpenReview, Hugging Face, Papers with Code, and Semantic Scholar produced 26 candidates:

Executed locally shuf -i 1-26 -n 1 returned 19, selecting QServe. No redraw was needed: paper, appendices, code, models, and benchmark materials were sufficient, and serving-kernel co-design does not duplicate recent PARQ training quantization or Q-VDiT video quantization.

2. Motivation: Why INT4 quantization can make execution slower

Quantization narratives usually begin with bandwidth and peak Tensor Core performance: A100's FP16, INT8, and INT4 Tensor Cores deliver 312, 624, and 1248 TOPS, with about 2 TB/s DRAM bandwidth (page 4, Figure 3). Those numbers suggest W4A4 should beat W8A8. QServe instead starts with measured 20%–90% dequantization overhead(abstract and page 2): INT4 weights or INT32 partial sums require scaling, zero-point handling, and type conversion on CUDA cores. On A100, one CUDA-core operation can cost roughly 50 INT4 Tensor Core operations(page 2). A few seemingly cheap scalar instructions in the main loop can thus stall faster matrix units.

The paper identifies two failure modes:

1. W4A16: Weights are stored in INT4 but GEMM uses FP16 Tensor Cores, requiring main-loop dequantization to FP16. This suits small-batch, memory-bound regimes. Roofline analysis finds W4A16 favorable on A100 around batch dimension m < 78 ; larger batches become limited by computation and dequantization (page 4, Figure 3).
2. W4A4: Group-wise activation quantization for accuracy requires repeated scaling and floating-point conversion of INT32 partial sums across K-dimension groups. Atom/QuaRot-style kernels show dequantization shares reaching 90% (page 14, Figure 18).

Serving sharpens the issue: decode KV cache grows linearly with sequence length and batch size, and attention accounts for 30%–50% of end-to-end time (page 10, §5.3). Four-bit weights with KV8/16 solve only half the memory wall; directly compressing KV to 4 bits adds bit operations and branches. Naive KV4 is 1.7× faster than KV8 on L40S but 1.2× slower on A100 (page 10). The actual systems question is how to reduce CUDA-core work in the weight path while keeping KV4 attention memory-bound.

3. Argument, assumptions, and trade-offs

QServe's narrative consists of three consecutive judgments:

• Judgment A: Precision combinations must follow the serving roofline. W4A16 suits small batches and W8A8 large batches; choose W4A8 to halve weight bandwidth while using INT8 Tensor Cores for matrix multiplication.
• Judgment B: The algorithm's format must be easy for the system to dequantize. QoQ is not arbitrary W4A8: per-channel INT8 is followed by group-wise INT4 within the INT8 domain, turning dequantization into register-level integer multiplication and subtraction.
• Judgment C: KV4 gains require attention-kernel support. SmoothAttention handles accuracy; bit tricks, simpler control flow, FP16 QK/SV, and asynchronous prefetching reduce arithmetic intensity back into the bandwidth-bound regime.

The tradeoff is explicit: W4A8KV4 is not the lowest arithmetic precision and has lower theoretical Tensor Core peaks than W4A4, but reduces serial CUDA-core work for higher effective throughput with moderate accuracy loss. Hardware dependence is strong: the paper recommends per-channel W4A8 on A100 and g128 group quantization on L40S with stronger CUDA cores (pages 12–13).

4. Formalization and procedure

For weight block (W), first-stage per-channel symmetric quantization uses the protected interval ([-119,119]):

[ s_c=\frac{\max |W_c|}{119},\qquad W^{(8)}_c=\operatorname{clip}!\left(\operatorname{round}(W_c/s_c),-119,119\right). ]

The second stage performs group-wise 4-bit asymmetric quantization:

[ q_{c,g}=\operatorname{clip}!\left(\operatorname{round}(W^{(8)}{c,g}/s{c,g})+z_{c,g},0,15\right), \quad \widehat W_{c,g}=s_c,s_{c,g}(q_{c,g}-z_{c,g}). ]

The key is not the generic two-stage shape, but [-119,119] the reserved overflow margin: QServe unpacks INT4 in parallel within registers, multiplies by integer scales, then subtracts precomputed zero-point compensation, avoiding FP16 dequantization in the GEMM main loop (pages 5–7, Figures 6–9).

Standard attention computes

[ O=\operatorname{softmax}(QK^\top/\sqrt d)V. ]

SmoothAttention introduces a per-channel scale (\lambda_j), setting (K'_j=K_j/\lambda_j, Q'_j=Q_j\lambda_j), preserving (Q'K'^\top=QK^\top). It estimates K magnitudes by channel and applies smoothing strength (\alpha), empirically effective at 0.5, moving difficult K outliers to Q, which requires no cached quantization (pages 7–8, Equations 5–7).

Evaluation pseudocode:

Input: FP16 model, calibration samples, target GPU
1  Search each block's rotations/scales and clipping to minimize block-output MSE
2  Quantize weights per-channel -> INT8[-119,119], then group-wise -> UINT4
3  Estimate K-channel magnitudes; move K outliers to Q with SmoothAttention
4  Select per-channel (A100) or g128 (L40S) for the target GPU
5  Reorder W4 packs offline to reduce address computation for adjacent Tensor Core data
6  During decode: unpack W4 in registers -> INT8; execute W4A8 GEMM on INT8 Tensor Cores
7  KV4 fused attention: bit tricks -> simplified control flow -> FP16 QK/SV -> async prefetch
Output: W4A8KV4 checkpoint and maximum-throughput serving engine

5. Key mechanisms and engineering details

6. Datasets, benchmarks, and metrics

• WikiText2 perplexity: Sequence length 2048, lower is better; covers Llama-1/2/3, Mistral, Mixtral, and Yi. Measures language-model fit, not chat, code, safety, or long-horizon factual consistency.
• Zero-shot tasks: The paper also uses LAMBADA, HellaSwag, PIQA, and WinoGrande, reporting mean accuracy in Table 3; main conclusions still emphasize perplexity and throughput.
• LongBench: v3 adds 10 Llama-3.1-8B-Instruct tasks; higher average is better (Table 5).
• Serving throughput: Single GPU, 1024 input tokens and 512 output tokens, reporting maximum tokens/s on A100-80GB-SXM4 and L40S-48GB. PyTorch 2.2.0, CUDA 12.2, TensorRT-LLM v0.9.0; baseline code as of 2024-04-18. Paged attention enabled (unsupported by QuaRot), in-flight batching disabled(pages 11–12). This is not SLO-constrained goodput and includes no TTFT/TPOT tail latency.

7. Complete main-results matrix

7.1 Accuracy: WikiText2 perplexity

The table reproduces the ten-model submatrix of paper Table 2 in full. Atom uses W4A4-g128 non-gray rows (not calibrated on WikiText2); AWQ/QoQ use W4A8KV4-g128. Bold marks the lowest value among these quantized methods; FP16 is a reference upper bound excluded from the “best quantized” designation.

QoQ's absolute perplexity improvements over same-format AWQ are, in order, 0.24, 0.16, 0.06, 0.05, 0.05, 0.13, 0.12, 0.09, 0.10, 0.05; approximate relative reductions are 3.46%, 2.74%, 1.17%, 1.42%, 0.84%, 2.43%, 2.73%, 1.64%, 2.36%, 1.05%(recalculated from Table 2). W8A8/W4A16 nevertheless remain closer to FP16 on most models, so QoQ cannot be called the most accurate; it optimizes the accuracy–memory–throughput tradeoff.

7.2 Throughput: L40S and A100

Unit: tokens/s. Each speedup uses the highest throughput among three TensorRT-LLM precisions as denominator (Table 4).

The strongest single result is Qwen1.5-72B on L40S: 17→59 tokens/s, +42 absolute and +247% (3.47×) relative. The A100 counterpart is 143→340, +197 and +138% (2.38×). The paper summarizes A100 gains as 1.2–2.4×, consistent with Table 4's columns. Across GPUs, L40S+QServe exceeds A100+best TRT on L3-8B, L2-7B, Mistral, L2-13B, and L1-30B; Yi-34B, 70B, and 72B either do not or lack comparable A100 baselines, so this cannot become “all models.”

7.3 LongBench

Paper Table 5: Llama-3.1-8B-Instruct, QoQ W4A8KV4-g128.

The average drops by just 0.14 points (−0.36% relative), with 4/10 tasks improving. However, this is one model and one evaluation without variance; increases cannot establish genuine capability gains from quantization.

8. Ablations, sensitivity, and transitions

Figure 16 gives the full chain for Llama2-7B, L40S, 64 requests, and 1024/512 tokens: W8A8KV8 baseline perplexity 5.58, throughput 1367; switching to W4 weights gives 6.00 and 1538 (1.12×, saving 3.5 GB). Block rotation reduces perplexity to 5.82, MSE clipping to 5.66; KV4 raises it to 5.80 but throughput to 2254 (1.47×, halving KV memory); SmoothAttention reaches 5.75; progressive group quantization 5.70 with throughput 2190; activation-aware reordering finally 5.67. Atom and QuaRot controls are 6.12/1732 and 6.19/688.

These ablations show that no single module explains all gains: accuracy recovery comes from rotation, clipping, SmoothAttention, and progressive grouping; speed primarily from W4 capacity and KV4 bandwidth. The final group/reorder stages trade roughly 64 tokens/s (2.84%) lower throughput for a 0.13 perplexity improvement (2254→2190, 5.80→5.67; recalculated here).

Figure 17 separates batch capacity from same-batch kernel speed: total Llama2-7B acceleration is 1.88× = 1.45× (same batch) × approximately 1.30× (larger batch); 13B's total is approximately 1.7×, split roughly equally. Figure 19's KV4 per-layer kernel latency falls from 0.48 ms→0.44 ms (bit tricks)→0.39 ms (simpler control)→0.36 ms (FP16 QK)→0.33 ms (FP16 SV)→0.28 ms (asynchronous prefetch), cumulatively 1.7×.

Hyperparameter sensitivity: The paper discusses SmoothAttention's (\alpha), finding 0.5 effective, and compares per-channel with g128. It supplies no complete cross-model numerical table, confidence intervals, or automatic selection rule. Thus hardware-dependent format switching is supported—per-channel on A100, g128 on L40S—but not global optimality of (\alpha=0.5) across models, layers, and lengths. The real phase transition comes from the roofline: larger batches move W4A16 from a bandwidth advantage into dequantization/compute bottlenecks; unoptimized KV4 similarly becomes compute-bound on A100.

Statistical significance: No repeated-throughput mean±standard deviation, tail latency, repeated perplexity experiments, or significance tests are reported. Small throughput gains such as 1.17× cannot be judged stable across driver versions/thermal states; 38.52→38.38 establishes only close point estimates, not statistical equivalence.

9. Related work: Problem origins and technical lineage

SmoothQuant (ICML 2023)

The paper Addresses activation outliers that make lossless W8A8 difficult: equivalent channel scaling moves quantization difficulty from activations to more easily quantized weights, reporting up to 1.56× speedup and approximately 2× memory reduction (its PDF, page 1). It remains at 8 bits and does not focus on KV cache. QServe inherits equivalent transformations that migrate quantization difficulty, but changes the destination from weights to uncached Q, protecting K with SmoothAttention. The inheritance is outlier migration → equivalent K/Q scaling.

AWQ (MLSys 2024)

The paper Identifies approximately 1% salient weights and protects them through activation-aware scaling, addressing W4 weight-only accuracy and edge speed; its PDF page 1 reports average 3.2–3.3× gains over Hugging Face FP16. W4A16 still needs FP16 Tensor Cores and online weight dequantization, potentially losing its advantage in large-batch cloud serving. QServe retains AWQ/clipping for accuracy but changes execution through W4A8 and a two-stage integer format. The inheritance is salient-aware weight protection → block MSE clipping + progressive integer dequantization.

Atom (MLSys 2024)

arXiv Targets throughput-oriented serving by quantizing weights/activations/KV cache while retaining a few outlier channels; its abstract reports up to 7.73× over FP16. The remaining issue is group-wise W4A4 activation partial-sum dequantization inside the GEMM main loop. QServe Figure 18 measures up to approximately 90% CUDA-core overhead, and Atom's public system supports only some models. QServe responds with W4A8, a protected range, and register-level INT4→INT8. The inheritance is end-to-end low-bit serving → avoiding floating-point partial-sum dequantization in the main loop.

QuaRot (NeurIPS 2024)

OpenReview Uses Hadamard rotation to remove hidden-state outliers and achieve end-to-end 4-bit weights, activations, and KV cache. As with Atom, accuracy-friendly group quantization does not necessarily yield high large-batch throughput; QuaRot lacked paged attention in the comparison. QServe retains rotation as accuracy preprocessing but selects W4A8KV4 and optimizes serving kernels. The inheritance is rotation removes outliers → rotation need not imply W4A4 execution.

vLLM / PagedAttention and TensorRT-LLM

vLLM SOSP 2023 Address KV-cache fragmentation and sharing through paging for higher serving throughput; TensorRT-LLM provides production-grade multiprecision kernels. Their remaining gap is not quantization theory but end-to-end kernels for the particular W4A8KV4 format. QServe adopts paged attention and maximum-throughput evaluation, contributing format–kernel co-design; disabled in-flight batching leaves scheduler-level benefits incompletely tested.

                         ┌─ SmoothQuant: equivalent scaling migrates activation outliers ─┐
FP16 serving ── W8A8 ───┤                                                            ├─ SmoothAttention: K→Q
   │                     └─ AWQ: activation-aware protection of salient weights ────────┤
   │                                                                                   ├─ QoQ W4A8KV4
   ├─ W4A16: saves weight bandwidth ── online FP16 dequantization bottleneck ───────────┤
   │                                                                                   │
   └─ End-to-end W4A4 ─┬─ Atom: groups + outlier channels ─┐                             │
                       └─ QuaRot: Hadamard removes outliers ┴─ FP partial-sum dequant bottleneck
                                                                                        │
vLLM PagedAttention ── KV paging/capacity ───────────────────────────────────────────────┤
TensorRT-LLM ── industrial baseline/multiprecision kernels ──────────────────────────────┘
                                      ↓
                         QServe: format × GEMM × KV4 fused attention

The turning point is a shift from model-file size or theoretical single-kernel peaks to the number of low-throughput instructions on the serving critical path. This is QServe's most reusable systems insight.

10. Contributions, reusable designs, and transfer potential

First, QServe makes non-Tensor-Core instructions a first-class constraint on precision selection. This transfers to FP4, MXFP4, MoE, and sparse kernels: format comparisons must count conversions, indexing, routing, and synchronization, not just MAC bit width.

Second, progressive group quantization demonstrates hardware-oriented intermediate representations: a protected range creates a parallelizable, overflow-safe INT8 intermediate before further compression. This could transfer to block-sparse weights, streaming expert weights, and mixed-precision caches.

Third, SmoothAttention is a clean structure-preserving transformation: move cached K outliers to transient Q without changing dot products. Abstractly, it moves error budget from long-lived state to short-lived computation, potentially applicable to cross-attention, visual-token caches, and RNN/SSM state quantization.

Fourth, Figure 17 separates throughput gains into same-batch kernel speed and larger-batch capacity, an attribution template other systems papers should reuse; otherwise capacity for more requests and faster operators collapse into one number.

11. Limitations and independent critique

Author-stated / documented limitations

1. Hardware dependence: Pages 12–13 explicitly use different quantization granularities on A100 and L40S; the Artifact Appendix primarily requires x86_64 and recommends A100/L40S. Results do not directly generalize to Hopper/Blackwell, AMD, or mobile devices.
2. Uneven baseline support: Page 12 notes unsupported models in Atom/QuaRot and missing paged attention in QuaRot; Table 4 therefore cannot provide a fully symmetric six-method matrix for every model.
3. Accuracy and configuration boundaries: Table 2 contains NaN for some methods/models, and gray rows use WikiText2 calibration. Calibration data can affect point estimates, so comparability must be checked row by row.

Independent critique (evidence → inference)

1. [External evaluation validity] Throughput is measured only with fixed 1024-input/512-output batch generation and in-flight batching disabled (§6.1), so better TTFT, TPOT, P99, or SLO goodput under online arrivals cannot be inferred.
2. [Baseline freshness] TensorRT-LLM is fixed at v0.9.0 and other branches at 2024-04-18 (page 12), whereas the final paper is from 2025 and verification occurs in 2026. The 1.17–3.47× gains are historical snapshots, not a current serving-framework ranking.
3. [Cost argument] “L40S $8K, A100 $25K, 3× lower cost” appears only as device prices in Figure 1, without cloud provider, purchase date, power, host costs, utilization, or depreciation. It supports an approximate GPU purchase-price ratio, not full TCO or 3× lower cost per qualified token.
4. [Statistical credibility] No throughput repeat counts, standard deviations, or significance tests are given. Small advantages such as A100 Llama2-70B's 1.17× may depend on frequency, temperature, drivers, and measurement noise; point estimates cannot prove stable superiority.
5. [Attribution fairness] QServe's total acceleration over TRT combines larger batch capacity and same-batch kernels (7B: 1.88×=1.45××approximately 1.30×, Figure 17). “The quantization kernel is 1.88× faster” is therefore incorrect; capacity and kernel gains must be reported separately.
6. [Quality coverage] The main quality matrix is WikiText2 perplexity; LongBench covers only one 8B model, with no chat-preference, coding, math, safety, factuality, or long-conversation stress tests. Average 38.52→38.38 cannot prove near-losslessness for all deployment tasks.
7. [Portability] Weight reordering, register parallelism, and KV4 bit tricks depend on A100/L40S Tensor Cores, CUDA cores, and register hierarchies; naive KV4 was 1.2× slower on A100. Algorithmic gains cannot be reproduced independently of specific kernels and GPU generations.
8. [Comparison completeness] Atom supports only Llama2-7B and QuaRot lacks paged attention; Table 4 fully lists only TRT and QServe. “Beats all SOTA systems” is not a head-to-head finding for most models; it beats runnable reported baselines.

12. Open challenges: Actionable head-to-head experimental gaps

Theory

• Format-selection model: On identical models/GPUs, compare W4A16, W8A8, W4A8KV4, and W4A4KV4, logging per-kernel Tensor Core utilization, CUDA-core instructions, register spills, and arithmetic intensity. Test whether a conversion-aware extended roofline predicts batch/length crossover points.
• Error propagation: Fix W4A8 weights and compare KV8, QoQ-KV4, and QuaRot-KV4 at 1k/8k/32k/128k lengths, measuring attention-logit KL, inter-layer state error, and task scores; determine whether K/V error grows linearly or undergoes transitions.

Engineering applications

• Online SLOs: Compare QServe with current TensorRT-LLM/vLLM on identical A100/L40S/H100 hardware under Poisson and bursty arrivals. Report TTFT/TPOT P50/P95/P99, goodput under 99% SLO, and energy/token rather than only maximum-batch throughput.
• Fair version-controlled retesting: Fix 2026 drivers, CUDA, compiler, and scheduler; integrate QoQ kernels into one runtime alongside FP8/INT8/INT4 backends, separating runtime-generation effects from format contributions.
• Multi-GPU: For 70B/72B on 1/2/4/8 GPUs, compare tensor/pipeline parallelism and KV-cache communication; determine whether all-reduce, dequantization, or load imbalance cancels KV4 savings.
• TCO: Measure dollars per million qualified output tokens on A100/L40S/H100 using actual cloud prices, power, and quality thresholds, upgrading the paper's 3× device-price claim into an auditable cost curve.

Emerging directions

• MoE W4A8KV4: Compare QServe with MoEQuant/FP8 on Mixtral/DeepSeek-style models, measuring expert-weight loading, routing imbalance, KV4 attention, and cross-device communication to identify whether weight bandwidth or expert communication becomes the bottleneck.
• Inference-time scaling: At equal answer accuracy, compare BF16, W8A8, and QoQ parallel sample counts and verifier costs. If quantization slightly reduces single-sample quality but enables more samples, assess final pass@k per dollar.
• Multimodal / world models: Compare QServe with KV8/FP8 on long visual-token VLMs and video KV caches, reporting visual grounding, temporal consistency, and throughput; test cross-modal validity of SmoothAttention's K→Q migration.

13. Recent-work radar (2025–2026)

1. TurboAttention, MLSys 2025: Combines attention approximation and quantization for over 4.4× KV-cache reduction; its paper page reports 1.2–1.8× attention speedup. This approximate alternative to QServe's exact KV4 kernel should be compared at equal quality thresholds.
2. FlashInfer, MLSys 2025: Unifies KV layouts and attention variants into customizable kernels, a natural foundation for testing QoQ integration into broader serving stacks.
3. LServe, MLSys 2025: Uses unified sparse attention to address quadratic long-sequence prefill and decode KV capacity; already merged with QServe in OmniServe, representing sparse × low-bit combinations.
4. LongRoPE2, ICML 2025: Expands context windows nearly losslessly but further increases KV-cache pressure; should be evaluated jointly with QServe/LServe, beyond needle retrieval alone.
5. RWKVQuant, ICML 2025: Extends quantization to RWKV's mixed scalar/vector representation, reminding us that QServe's KV-cache-centered narrative does not fit every sequence architecture.
6. SkipGPT, ICML 2025: Dynamically skips layers per token, reducing computation rather than bits per operation; combining it with QServe requires checking whether dynamic control flow shifts the bottleneck back to CUDA cores.
7. Cache Me If You Must, ICML 2025: Allocates KV quantization precision by importance. Comparing selectively retained high-precision KV with QoQ's uniform KV4 would clarify quality–memory–kernel-complexity tradeoffs.

14. Conclusion

QServe's strongest evidence is that under the same single-GPU memory budget, W4A8KV4 raises Qwen1.5-72B L40S throughput from the best TensorRT-LLM's 17 to 59 tokens/s (3.47×), and A100 from 143 to 340 (2.38×). Its deeper contribution is showing that low-bit success depends on CUDA-core instructions on the critical path: naive KV4 is 1.2× slower on A100 and becomes approximately 1.5× faster only after optimization.

However, “3× lower cost” should be read as an illustration based on particular device prices, not a TCO conclusion. Fixed lengths, disabled in-flight batching, older TensorRT-LLM, absent tail latency/standard deviations, and single-GPU scope limit production extrapolation. Its reusable research methodology is to jointly report format accuracy, capacity-driven batch gains, same-batch kernel gains, and hardware crossover points.

Generation and validation notes: The core PDF, four technical-source papers, and every listed URL were actually accessed/read. The candidate pool is stored alongside this report in candidates_2026-08-28.tsv. The HTML uses the Warm Editorial single-file template with static HTML/CSS/JS validation. The runtime lacked a local Chromium for /usr/bin/chromium or /opt/pw-browsers , preventing honest completion of Playwright light/dark/mobile screenshots. Responsive rules, horizontal table containers, theme variables, and DOM checks were used instead; no browser rendering is falsely claimed. ml-report-index.md has been appended with this record.