Seesaw: Different Parallelism Configurations for Prefill and Decode

Date: 2026-09-25
Field: Efficiency and Systems (distributed inference, parallelism strategies, KV cache)
Core paper: Seesaw: High-throughput LLM Inference via Model Re-sharding，MLSys 2025
Authors: Qidong Su, Wei Zhao, Xin Li, Muralidhar Andoorveedu, Chenhao Jiang, Zhanda Zhu, Kevin Song, Christina Giannoula, Gennady Pekhimenko
Evidence convention: All paper-reported numbers below come from the PDF actually read. “PDF p.x” refers to downloaded-file pagination; “Fig./Table/§” follows paper numbering. Charts without numerical labels are not estimated from pixels.

0. Link verification table and version boundaries

All links were actually visited on 2026-09-25 . The core paper’s OpenReview page triggered browser verification, but its PDF URL was accessible; OpenReview is therefore marked as having a verification-restricted entry page and is not used as the source of numerical results. No official Seesaw code, data/weights, project page, or video was found in the paper, official conference page, arXiv, or searchable repositories; each is explicitly marked “not released/not found.”

1. Candidate pool, deduplication, and random selection

The run date is Friday, so the field rotates to “Efficiency and Systems” under the rules. First checked ml-report-index.md: the preceding 14 days already covered Search Your Block Floating Point Scales!(low-bit quantization),Ada-KV(KV cache allocation), and, slightly earlier, Lumos(training-performance modeling). These three papers and their topics were excluded. Candidates were collected from MLSys 2025 proceedings and official OSDI 2025 pages, with arXiv, OpenReview, Hugging Face Daily Papers, Papers with Code, and Semantic Scholar used for cross-discovery. Some aggregators supplied no new primary-source version information, so final links consistently point to official conference entries. The following 23official entries were all visited and returned usable pages:

1. Youmu: Efficient Columnar Data Pipeline for LLM Training — MLSys 2025
   
2. LeanAttention: Hardware-Aware Scalable Attention Mechanism for Decode — MLSys 2025
   
3. Rethinking Key-Value Cache Compression Techniques for LLM Serving — MLSys 2025
   
4. SampleAttention: Near-Lossless Long Context LLM Inference — MLSys 2025
   
5. ReaL: Efficient RLHF Training with Parameter Reallocation — MLSys 2025
   
6. APOLLO: SGD-like Memory, AdamW-level Performance — MLSys 2025
   
7. PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training — MLSys 2025
   
8. HyC-LoRA: Memory Efficient LoRA Fine-tuning with Hybrid Activation Compression — MLSys 2025
   
9. XGrammar: Flexible and Efficient Structured Generation Engine — MLSys 2025
   
10. FlexAttention: A Programming Model for Generating Fused Attention Variants — MLSys 2025
    
11. NEO: CPU Offloading for Online LLM Inference — MLSys 2025
    
12. Context Parallelism for Scalable Million-Token Inference — MLSys 2025
    
13. Marconi: Prefix Caching for the Era of Hybrid LLMs — MLSys 2025
    
14. FastTree: Attention Kernel and Runtime for Tree-Structured LLM Inference — MLSys 2025
    
15. Scaling Deep Learning Training with MPMD Pipeline Parallelism — MLSys 2025
    
16. Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking — MLSys 2025
    
17. Optimizing LLM Queries in Relational Data Analytics Workloads — MLSys 2025
    
18. SOLA: SLO Attainment with State-Aware Scheduling — MLSys 2025
    
19. ThunderServe: High-performance Cost-efficient LLM Serving in Clouds — MLSys 2025
    
20. Seesaw: High-throughput LLM Inference via Model Re-sharding — MLSys 2025
    
21. LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention — MLSys 2025
    
22. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — MLSys 2025
    
23. WaferLLM: Large Language Model Inference at Wafer Scale — OSDI 2025

After excluding the previously covered Lumos (original number16) from the initial24-item list, actually executed printf '%s\n' {1..15} {17..24} | shuf -n 1, returning original number 21, namely Seesaw. Its official PDF contains complete models, hardware, workloads, end-to-end results, and sensitivity analyses. The material is sufficient, sozero redraws。

2. Motivation: why static parallelism is structurally mismatched

The issue is not whether TP or PP is better overall, but that the dominant bottlenecks switch between phases. Prefill processes hundreds to thousands of input tokens at once, amortizing weight loading over many tokens; compute and all-reduce communication dominate runtime. Decode generates only one token per sequence per step, has low arithmetic intensity, and spends a larger fraction of time loading layer weights from HBM. TP shards linear-layer weights across GPUs, enabling parallel loading during decode, but synchronizes full activations at every layer. PP communicates activations only across stages, reducing prefill communication, but divides the global batch into microbatches of roughly 1/PP , requiring PP times as many forward passes for the same global batch and repeatedly loading weights. Fig.3 explicitly illustrates these opposing preferences (PDF pp.4–5).

Three sets of numbers substantiate the problem. First, low-bandwidth links turn TP’s per-layer all-reduce into a prefill tax: experimental PCIe4.0×8 bandwidth is only 16 GiB/s, versus NVLink’s 600 GiB/s(§6.1, PDF p.8). Second, spatially separating prefill/decode does not automatically solve small-cluster limitations: LLaMA2-70B FP16 weights occupy about 140 GiB, requiring at least 4 GPUs per phase on 8×40GiB GPUs and forcing a4/4 split. In that configuration, prefill throughput exceeds decode by more than 6×, while 4-GPU decode throughput is only 15%of 8-GPU decode throughput (§3.2, Fig.4, PDF p.6). Third, real A100 comparisons show that connectivity determines the gain: on NVLink, Seesaw still improves70B ShareGPT by 13% , but on PCIe the gains for arxiv-summarization and ShareGPT are respectively 46% with 30%（§6.4、Fig. 11，PDF p.9）。

The core assumption is therefore that the same physical GPUs can be time-multiplexed into different logical parallel topologies, provided resharding and KV migration costs are amortized over larger phase batches or hidden asynchronously. This naturally favors offline/high-throughput workloads. Under strict TTFT/TPOT requirements, accumulating CPU KV buffers until full adds waiting; the authors explicitly acknowledge trading latency for throughput in§7.1 (PDF p.10).

3. Narrative, formalization, and key tradeoffs

3.1 Mean time-per-sequence model

In§3.1 (PDF p.5), the paper expresses average time per sequence as:

$$T \approx \frac{T_{dm}^{linear}}{TP} + \frac{T_{dm}^{attn}+T_{comp}}{DP\cdot TP\cdot PP} + \frac{T_{comm}(TP)}{PP\cdot DP}.$$

where T_dm^linear mainly represents linear-layer weight movement,T_dm^attn mainly represents KV-cache movement,T_comp is computation, andT_comm(TP) is communication, increasing monotonically with TP. The first term dominates decode, favoring higher TP; it is relatively negligible in prefill, where the third term matters more, favoring lower TP and higher PP/DP. The model turns parallelism selection from a heuristic into a tradeoff among three resource terms. Its limitation is that switching, CPU-memory bandwidth, NUMA, and PCIe contention are absent from the main equation and handled afterward by system mechanisms.

3.2 Dynamic resharding flow

Incoming request queue
   │
   ▼
Select prefill configuration c_p (often PP-heavy, fewer all-reduces)
   │  Process prompts; GPU KV → pinned memory → shared CPU memory
   │  Repeat until the CPU KV buffer is full
   ▼
Reload target weight shards from CPU and retrieve KV shards under c_d
   │
   ▼
Select decode configuration c_d (often TP-heavy, parallel weight loading)
   │  Prefetcher swaps in asynchronously; decode with continuous batching
   │  Repeat until the CPU KV buffer is empty
   └───────────────────────────────↺ Return to prefill

The following pseudocode is closer to implementation. It is not verbatim paper code, but a faithful procedural rendering of§4–§5:

while pending_requests or live_sequences:
    if cpu_kv_store.is_empty():
        reshard_weights(target=c_p)
        while not cpu_kv_store.is_full() and pending_requests:
            batch = schedule_prefill(pending_requests)
            kv = workers.prefill(batch, layout=HND)
            async_swap_out(kv, gpu_to_pinned=True, pinned_to_shared=True)
    else:
        reshard_weights(target=c_d)
        while not cpu_kv_store.is_empty():
            prefetcher.async_swap_in(free_gpu_kv_slots, shards_for=c_d)
            batch = schedule_decode(ready_sequences)
            workers.decode(batch)

The tradeoffs are real. Fewer phase switches mean larger batches and higher throughput, but longer request waits. CPU KV buffering expands prefill batches but makes CPU-memory capacity, NUMA paths, and PCIe bandwidth new constraints. Reloading weight shards from CPU costs roughly the same regardless of batch size and can be amortized, whereas KV movement grows with batch size and must be hidden through overlap (§5.2, PDF p.7).

4. Key mechanisms and engineering details

Not changing DP dynamically is an important design boundary. Changing DP alters model-replica occupancy and the share of GPU memory available for KV, either increasing complexity or adding CPU↔GPU transfers. The authors therefore adjust only TP/PP dynamically (§4.1, PDF p.6). This keeps implementation manageable but means “dynamic parallelism” is not an automatic search over the entire space.

5. Datasets, models, hardware, and metrics

The authors exclude SGLang and DeepSpeed-FastGen because they do not support PP, and TensorRT-LLM because its scheduling resembles vLLM’s and vLLM has comparable throughput (§6.1, PDF p.8). This explains baseline selection but does not replace head-to-head comparisons with DistServe, Sarathi-Serve, or newer serving stacks.

6. Main results: what the paper establishes

6.1 Complete readable cross-hardware summary

All relative gains use vLLM on the same hardware as the denominator.1.45× This can be expressed either as 45% above baseline or a throughput multiplier of 1.45. Both forms are retained to avoid confusing pp with percentages. Throughput is not a percentage rate, so percentage points are inappropriate.

6.2 A100 normalized matrix: where the paper provides exact bar values

Fig.11 normalizes across interconnects to “vLLM + NVLink =1.00.” The paper has only two real datasets, so it cannot honestly be expanded into five benchmarks×six methods. Below is the complete directly readable 2×4 matrix, with missing information preserved as missing.

Direct calculations from bar values give, on PCIe, arxiv 0.89/0.61−1=45.90%and ShareGPT 0.82/0.62−1=32.26%; the main text reports 46% and 30%, possibly because plotted values are rounded to two decimals. To avoid spurious precision, subsequent conclusions retain the main text’s 46% / 30%. Similarly, vLLM PCIe achieves about 60%of NVLink throughput, while Seesaw raises this to 82%–89%. Its main value is therefore approaching fast-interconnect throughput in a low-bandwidth environment, rather than outperforming fast interconnects (§6.4, PDF p.9).

6.3 Why there is no five-benchmark×six-method matrix

The paper publishes only two real datasets, one vLLM baseline, and several fixed parallel configurations. Fig.10 does not print exact values for individual model/dataset bars, providing only aggregate speedups. Filling a matrix by estimating bar heights would invent precision. Thus, explicitly:the paper does not report an auditable five-benchmark×six-method absolute-throughput matrix, nor a complete tokens/s or requests/s table. We can confirm superiority over the best fixed configuration in the target setting, but cannot compute per-GPU cost, energy, or cluster TCO.

7. Layered ablations, sensitivity, and phase transitions

7.1 Mechanism-level breakdown

Fig.12 (PDF p.9) decomposes prefill, decode, and mixed batches using CodeLLaMA-34B,4×A10, and arxiv-summarization. PP4 has the fastest prefill, TP4 the fastest decode, and Seesaw uses P4→D4 to approach both phase optima. The best static vLLM configuration is TP2PP2 with chunked prefill. The authors attribute the gap to two factors: chunked prefill still leaves pure decode steps, and TP2PP2 prefill is slower than PP4.However, Fig.12 provides no exact milliseconds or throughput labels on its bars, so the individual contributions of resharding, tiered KV, and asynchronous transfers cannot be quantified, nor can any component’s share be claimed.

The paper has no standard component-by-component full → −async → −tiered → −reshard ablation table. What is observable is a comparison of system paths, not causal isolation. It supports the combined system outperforming the best static configuration, but not claims that the tiered buffer alone contributes X% or the asynchronous pipeline hides Y% of overhead.

7.2 Phase transition with input/output ratio

Fig.13 (PDF pp.9–10) fixes input length at 3,000 tokens for 70B on 8×A10 and varies output length. At only 1 output token, PP8 and Seesaw are similar, while TP4PP2 is slower due to communication. As output grows, PP8’s repeated decode weight-loading cost emerges; the best fixed configuration shifts first to TP2PP4 and finally to TP4PP2 when decode dominates. Seesaw exceeds fixed strategies throughout the ratio range. The actual transition is from prefill communication dominance, through a mixed regime, to decode weight-bandwidth dominance—not a monotonic benefit from a magic hyperparameter.

Exact values are not printed for every length point, so this report does not estimate the output-token count at each crossover. Only strategy order and qualitative trends can be confirmed. The authors infer that dynamic adaptation would help further when variable-length real requests share a batch, but provide no separate significance experiment with randomized length distributions to establish that inference.

7.3 Interconnect-bandwidth sensitivity

Fig.14 (PDF p.10) covers PCIe bandwidth from 0.1×–50×: PP is best at low bandwidth, TP at high bandwidth, and Seesaw exceeds fixed strategies throughout. Crucially, these are not measurements on multiple0.1×,1×, and 50× networks. The authors measure and trace all-reduce on 34B,8×A10, and the arxiv workload, then mutate all-reduce time to project throughput. The result therefore supports the model’s expected direction under bandwidth changes, not real end-to-end evidence across networks.

7.4 Hyperparameters and statistical significance

The paper reports tuning vLLM chunk size, but omits the full search grid and final chunk size for each combination. It also omits sensitivity to CPU KV buffer thresholds, prefetch depth, and pinned-buffer count. Total CPU KV capacity affects switching frequency and waiting time, yet receives no separate sweep.

Statistically, the paperNot reportedrandom seeds, repeat counts, means±standard deviations, confidence intervals, error bars, or significance tests. System throughput often has low variance, but that does not replace reporting. In particular, single-point charts cannot establish whether the 13% NVLink gain is robust or cloud-instance noise changes rankings. Thus,1.36× is an observed experimental aggregate, not a population-effect estimate with a confidence interval.

8. Related work: problem origins and technical lineage

8.1 vLLM / PagedAttention: reframing KV fragmentation as paging

Problem solved. vLLM partitions KV cache into fixed blocks through PagedAttention, enabling near-zero waste and cross-request sharing. Its abstract reports 2–4× throughput over FasterTransformer/Orca at the same latency (vLLM PDF p.1). It provides the foundation of continuous batching and efficient KV management, and is Seesaw’s only main baseline.

Remaining gap. PagedAttention addresses KV capacity and fragmentation, but not the opposing prefill/decode objectives of one static TP/PP configuration. Larger batches remain constrained by the tradeoff between communication and weight bandwidth under a fixed topology.

How Seesaw responds. Seesaw directly reuses vLLM model implementations and sweeps its fixed parallel configurations, then extends the optimization axis from paged KV management to phase-level topology changes.

Design transfer. vLLM’s blocks and continuous batching inform Seesaw’s GPU KV-slot management; Seesaw then extends KV ownership from a single GPU layout to an intermediate state in shared CPU memory.

8.2 DistServe: separate GPUs for the two phases, targeting goodput/SLOs

Problem solved. DistServe spatially separates prefill and decode, independently plans resources and parallelism, and targets goodput under TTFT/TPOT constraints. Its abstract reports serving 7.4× more requests or tightening SLOs by 12.6×while more than 90% of requests meet constraints (DistServe PDF p.1).

Remaining gap. Spatial separation requires each phase to hold the model, giving coarse resource granularity. Seesaw’s70B example shows that 8×40GiB permits only a4/4 split, making prefill over 6× faster than decode, while 4-GPU decode reaches only 15% of 8-GPU throughput.

How Seesaw responds. It separates phases in time: all GPUs perform prefill together, then decode together. Small clusters thus avoid a permanent4/4 resource allocation.

Design transfer. DistServe establishes phase heterogeneity as a system boundary; Seesaw turns that boundary into switchable model-shard layouts. Their objectives differ: DistServe prioritizes SLOs, Seesaw offline throughput.

8.3 Sarathi-Serve: chunking prefill into decode instead of splitting the cluster

Problem solved. Sarathi-Serve combines chunked prefill with stall-free batching to create uniform iterations and reduce pipeline bubbles. Its abstract reports serving-capacity gains for Mistral-7B on one A100 of 2.6×, Yi-34B on two A100s of 3.7×, and Falcon-180B under PP of up to 5.6×（Sarathi PDF p.1）。

Remaining gap. Large chunks still leave decode-only steps; small chunks reduce kernel efficiency. Fixed TP/PP must still serve both phases. Seesaw§7.1 explicitly discusses this chunk-size dilemma.

How Seesaw responds. The baseline already enables and tunes chunked prefill; Seesaw still uses a TP configuration for pure decode steps and PP for prefill. It builds phase resharding on top of the remaining fixed-topology limitation rather than bypassing Sarathi.

Design transfer. Sarathi’s explicit phase scheduling informs Seesaw’s scheduler. A natural future combination would use chunked prefill for fine-grained latency/throughput balance and resharding for the parallel topology of remaining pure phases.

8.4 Mooncake: a global KVCache pool spanning CPU/DRAM/SSD/NIC

Problem solved. Mooncake builds KVCache-centric global caching and scheduling for spatially separated prefill/decode. Its arXiv v4 abstract reports simulated throughput gains up to +525%and handling 75% more requests on real workloads (Mooncake PDF p.1); the official FAST page separately reports real-trace capacity gains of 59%–498% and deployment across thousands of nodes.

Remaining gap. It assumes a large serving cluster, complex storage/network layers, and SLO scheduling; small commodity-GPU nodes may not support that architecture.

How Seesaw responds. Seesaw uses only shared CPU memory as an intermediate state for phase switching, targeting limited GPUs and slow links in a single node/small cluster. This sacrifices online latency while substantially improving resource-allocation granularity.

Design transfer. Both Mooncake and Seesaw show that KV is now scheduling state, not merely an internal GPU cache. Mooncake makes it a cross-node storage object; Seesaw makes it a medium for resharding across parallel layouts.

9. Nonlinear technical evolution and cross-paper comparison

                                                            ┌─ DistServe ── Spatial P/D separation, independent SLO planning
Phase heterogeneity ──────────┤
                              └─ Mooncake ─── Global KVCache pool, cross-node reuse
                                      │
Manageable KV memory ─ vLLM/PagedAttention ──┼──────────────┐
                                      │              │
Smoother scheduling ─ Sarathi-Serve ─ chunked prefill │
                                      │              ▼
                                      └──────► Seesaw
                                         Time-multiplex all GPUs
                                  PP-prefill ⇄ TP-decode
                                      │
                                      ├─ tiered CPU KV
                                      ├─ async resharding
                                      └─ transition-minimizing schedule

Three shifts drive this evolution. First, from fitting model weights to eliminating KV waste: vLLM expands batches. Second, from one request pipeline to heterogeneous prefill/decode resources: DistServe/Mooncake make phases separate scheduling units. Third, from dedicating different GPUs to phases to changing the topology of the same GPUs over time: Seesaw trades switching cost for finer resource allocation. Sarathi follows a separate branch, reducing interference by combining rather than separating phases. Seesaw shows that fixed topologies retain losses even with chunked prefill.

These multipliers cannot rank which system is best because baselines, models, hardware, SLOs, and metrics differ. The table compares design goals and evidence forms, not cross-paper leaderboard positions.

10. Actual contributions, reusable designs, and transfer potential

Seesaw’s contribution is neither CPU offloading alone nor another parallelism searcher. It incorporates parallel layout into the request lifecycle: topology becomes phase-dependent rather than a static deployment decision. This abstraction directly exploits the fact that prefill is more communication-sensitive and decode more weight-bandwidth-sensitive.

Three especially reusable designs are: a complete CPU KV image as a canonical intermediate representation between shard layouts, avoiding complex GPU all-to-all; separating GPU→pinned from pinned→shared transfers to work around non-pinnable shared memory; and coupling switch counts to batch size through CPU KV full/empty conditions, amortizing overhead over phase batches. These can transfer to inference engines, heterogeneous accelerators, and serving orchestration.

With quantized models, smaller weights reduce decode’s weight-bandwidth share and alter TP’s benefit, requiring reselection of c_d. With MoE, expert routing and all-to-all may replace all-reduce as the main bottleneck, and CPU reloading of expert shards becomes more complex. With speculative decoding, processing several draft tokens per decode raises arithmetic intensity and weakens the prefill/decode distinction. With prefix caching, the CPU KV store must maintain shared-prefix references and consistency. Thus, phase adaptation transfers, but the paper’s specific PP→TP strategy cannot simply be copied.

11. Limitations acknowledged by the paper

1. Throughput first, at the expense of latency. The authors explicitly state that chunked prefill can serve both throughput and latency, whereas Seesaw prioritizes throughput at the cost of increased latency (§7.1, PDF p.10). Conclusions therefore cover offline use and should not be extrapolated to interactive chat.
2. Most favorable under low bandwidth/resource constraints. A10’s mean 1.45× exceeds L4’s 1.29×, while ShareGPT on NVLink retains only 13%; the authors therefore target commodity GPUs and slow interconnects (§6.2,§6.4–6.5, PDF pp.8–10).
3. DP is excluded from the dynamic space. Because changing DP alters replica occupancy and KV-memory shares and adds movement, only TP/PP changes dynamically (§4.1, PDF p.6). Dynamic model resharding is therefore not a global optimizer over arbitrary parallel dimensions.

12. Independent critique: evidence-to-inference chains

1. [Comparison fairness] Because the main experiments compare only with vLLM0.5.4, without same-hardware head-to-head tests of DistServe, Sarathi-Serve, Mooncake, or TensorRT-LLM (§6.1, PDF p.8), 1.36× means a gain over this tuned vLLM version, not superiority over all contemporary systems。
2. [Reproducibility] Because no official Seesaw code, configuration files, or reproduction scripts were found by the verification date, and the paper omits final per-workload chunk sizes and all c_p/c_d choices, independent researchers cannot easily verify whether the 1.06–1.78× range depends on hidden tuning。
3. [Metric completeness] Because end-to-end throughput is the only main metric, with no absolute requests/s, tokens/s, TTFT, TPOT, P50/P99, or SLO attainment, the 36% average relative gain cannot be translated into capacity planning, user experience, or value per dollar。
4. [Causal attribution] Because Fig.12 lacks component-level values for −tiered buffer、−async pipeline、−transition minimization and compares only the complete system with fixed parallel paths, the combined design is effective, but resharding selection’s contribution relative to supporting mechanisms remains unknown。
5. [External validity] Because evaluation includes only 15B/34B/70B FP16 dense GQA models, excluding MoE, quantization, speculative decoding, and prefix caching—which alter weight bandwidth, KV share, and communication— the PP-prefill/TP-decode ranking cannot be assumed to persist in modern inference stacks。
6. [Resource accounting] Because experiments allocate each GPU 80 GiB CPU memory, an8-GPU node requires 640 GiB of host memory, while DRAM/NUMA/CPU-copy bandwidth and energy are excluded from cost, the claim that commodity GPUs are more economical may merely shift costs to the host。
7. [Bandwidth sensitivity] Because the 0.1×–50× PCIe curve projects throughput by modifying traced all-reduce time rather than running on multiple networks, it validates model-based extrapolation, not end-to-end portability under congestion, NUMA, and DMA contention。
8. [Statistical reliability] Because the paper gives no repeat counts, error bars, or significance tests, the moderate13% NVLink gain in particular cannot be rigorously separated from cloud-instance jitter, thermal state, and scheduling noise。

13. Actionable open challenges

13.1 Theory

Experiment gap T1: optimal online switching. Under identical arrivals and hardware, compare (A) Seesaw’s full/empty thresholds、(B) an MDP policy using remaining KV/estimated output length、(C) oracle dynamic programming; sweep arrival rate and output-length coefficient of variation, reporting throughput, mean waiting time, P99 TTFT, switch count, and transferred bytes. Estimate how far Seesaw’s threshold policy lies from the Pareto frontier.

Experiment gap T2: an analytical model including resharding cost. Compare the original equation, an extended model with CPU-bandwidth/NUMA/resharding terms, and measurements. Run a full-factorial experiment over c_p×c_d×batch×CPU-KV-size on A10/L4/A100, evaluated by MAPE and top-1 configuration accuracy. Unless the extended model selects configurations better, its terms cannot be claimed to explain system gains.

13.2 Engineering applications

Experiment gap E1: fair comparison within one stack. Use the same vLLM backend version, tokenizer, prompt/output trace, and 8-GPU node to compare Seesaw, tuned vLLM chunked prefill, Sarathi-Serve, and DistServe. Report tokens/s, requests/s, P50/P99 TTFT/TPOT, GPU/CPU memory, PCIe bytes, energy, and failure recovery. This establishes the net benefit of resharding versus spatial separation and phase merging.

Experiment gap E2: CPU resource stress tests. Fix GPUs and workload; sweep per-GPU CPU KV budgets of 10/20/40/80/160 GiB, single/dual NUMA, and PCIe×4/×8/×16. Compare Seesaw with the best static configuration without a CPU tier. If gains vanish below 80GiB, deployment guidance must explicitly require that host-memory capacity.

Experiment gap E3: modern inference combinations. Repeat the c_p/c_d search separately for FP16, INT8, INT4, MoE, and EAGLE speculative decoding, comparing static configurations, Seesaw, and Seesaw+chunked prefill. Test whether switching remains beneficial as decode arithmetic intensity rises, making changes in optimal strategy a primary result rather than reporting throughput alone.

13.3 Emerging directions

Experiment gap N1: cross-request prefix/KV reuse. compare Seesaw-HND full copy、copy-on-write shared prefix、Mooncake-like global KV object, using traces with 0%/25%/50%/75% prefix sharing. Measure transferred bytes, hit rate, throughput, first-token latency, and consistency errors. This tests whether complete CPU KV images repeatedly transfer shared prefixes.

Experiment gap N2: multitenancy and elastic GPUs. Compare static vLLM, Seesaw, and DistServe under preemption, GPU failures, and changing tenant load. Measure resharding recovery time, KV loss, tail latency, and effective throughput. Seesaw’s CPU intermediate state could anchor recovery or create a single-machine failure domain; distinguish them with fault injection.

14. Recent-work radar

1. ThunderServe（MLSys 2025）: extends single-node resharding toward heterogeneous cloud instances and cost-aware scheduling; useful for testing whether within-node topology adaptation and between-node resource adaptation can combine.
2. LServe（MLSys 2025）: reduces long-sequence KV/compute pressure with unified sparse attention; shrinking attention costs would shift the phase balance in Seesaw’s analytical model.
3. FlashInfer（MLSys 2025）: supplies composable attention kernels and scheduling primitives to serving stacks; potentially reduces kernel losses in fixed configurations and provides a stronger backend control.
4. NEO（MLSys 2025）: CPU offloading for online inference directly contrasts with Seesaw’s use of CPU only as an intermediate state, testing whether CPU computation is better than pure transfer.
5. SOLA（MLSys 2025）: targets SLO attainment through state-aware scheduling, representing the state and tail-latency control Seesaw needs to move online.
6. WaferLLM（OSDI 2025）: studies LLM inference on wafer-scale interconnects. Seesaw predicts TP dominance at very high communication bandwidth, and this work supplies a contrasting hardware reference.
7. Context Parallelism for Scalable Million-Token Inference（MLSys 2025）: long contexts make KV/attention dominant, potentially requiring CP in Seesaw’s currently TP/PP-only dynamic space.

15. Conclusion

Seesaw elevates parallelism selection from a deployment parameter to inference-phase state. On low-bandwidth small clusters, prefill uses communication-friendly PP and decode weight-bandwidth-friendly TP, with CPU KV tiers and asynchronous copies hiding switching costs. On A10, the paper reports 1.45× geometric mean and 1.78× maximum, with an overall A10+L4 mean of 1.36×; on A100 PCIe, it raises vLLM’s roughly 60% of NVLink throughput to 82%–89%（Fig. 10–11，PDF pp.8–9）。

The most defensible conclusion is narrower: this is an effective combined design for offline, throughput-first workloads with low-bandwidth interconnects , not a comprehensive replacement for online serving. Missing code, absolute throughput, latency/SLO metrics, component ablations, modern model combinations, and statistical intervals prevent1.36× from becoming a production-capacity commitment. The key next experiment is a direct comparison with DistServe, Sarathi-Serve, and modern vLLM on the same stack, trace, and latency constraints.

Index status: Updated ml-report-index.mdwith this report’s date, field, title, venue, and one-sentence conclusion.
Source notes: The official MLSys page and arXiv both list Wei Zhao; font extraction from the downloaded PDF misreads the name as “Wei hao.” This report follows official metadata. Unverified code, data, project pages, and the core paper’s video are explicitly marked; third-party mirrors were not substituted for official sources.
Visual verification notes: Playwright was actually invoked, but neither the local /usr/bin/chromium、/opt/pw-browsers nor Playwright cache contains a Chromium executable. Under the constraint against running playwright install, no browser was installed over the network, so real light/dark/mobile screenshot checks could not be completed. The Warm Editorial deterministic validator (0 warnings), HTML semantic checks, and static responsive-CSS review were used instead; these do not fully replace browser pixel-level verification.