RSD: Letting Process Rewards Decide When Large Models Take Over

0. Link verification table

All URLs below were visited. Experimental numbers come only from the official ICML/PMLR version, arXiv v3 PDF, and official implementation. Hugging Face, Papers with Code, and Semantic Scholar support discovery or cross-checking, not numerical claims. OpenReview metadata was located, but automated access encountered a browser challenge, so its contents are not evidence here.

1. Candidate pool, deduplication, and random selection

The run date was determined by date +%u returned 5, assigning this round to efficiency and systems. First checked ml-report-index.md: the preceding 14 days covered SuperInfer, LRAM, RGM/HPS/ReActR, and related topics; earlier reports covered Ada-KV and NVFP4 scale search. To avoid repeating conclusions with different models, KV eviction, recurrent backbones, and block floating-point quantization were excluded as core topics.

Candidates came from official ICML/NeurIPS/MLSys/ICLR proceedings or accepted-paper pages, supplemented and deduplicated through arXiv, OpenReview, Hugging Face Daily Papers, Papers with Code, and Semantic Scholar. The table lists the 24 actual random-pool candidates. Where individual URLs are absent, it links the visited official conference entry rather than guessed unvisited links.

Actually executed shuf -i 1-24 -n 1returned 18. RSD is an ICML 2025 main-conference paper within 24 months, with complete official text, appendix, and code, and no overlap with the preceding 14 days' core topics; therefore adopted on the first draw, without redrawing. Random selection determines what to read, not the strength of the conclusions.

2. Motivation: Exact Token Matching Misses Semantic Redundancy in Reasoning Steps

Standard speculative decoding lets a small draft model propose tokens, then a large target verifies them in parallel. Its key advantage is unbiased output: rejection-sampling correction preserves the target distribution exactly. But acceptance is strict. A draft saying “therefore the answer is 4” may be rejected when the target prefers “so the result is 4,” despite equivalent reasoning, if token probabilities do not match. For long chains of thought, this can discard correct work already completed by the small model.

Table 2 provides direct evidence: Qwen2.5-Math-7B target averages 65.3across six mathematical/scientific benchmarks, while token-level SD with a 7B target, 1.5B draft, and 7 speculative tokens achieves 64.6; RSD with a 7B PRM and fixed threshold 0.7 reaches 66.1(PDF p.5, Table 2). Theoretical unbiasedness does not guarantee higher finite-sample accuracy. RSD deliberately gives up exact target-distribution preservation in exchange for semantic variation accepted by process rewards.

A second problem is best-of-N. Sampling 16 full reasoning traces independently from a 1.5B model and selecting with a 7B reward model averages 64.8in the same table, below RSD's 66.1. On MATH-500, the 1.5B model, Process Best-of-8, Best-of-16, Beam-4, Beam-8, and RSD score respectively 73.8、75.8、76.0、78.2、78.2、82.6(PDF p.6, Table 3). Search increases coverage but applies rewards at sequence endpoints or across width; RSD instead uses rewards to route each reasoning step.

A third issue is where test-time computation is spent. Difficult problems need large models, but easy steps may not. By MATH difficulty, at threshold 0.7 the share of entirely draft-completed problems falls from Level 1's 84% to Level 5's 19%; the accuracy gain from target intervention rises from Level 1's +4.7 pp to Levels 4/5's +16.4/+15.7 pp(Appendix B.2, PDF p.17, Figure B.1). This supports reserving expensive models for difficult steps, while emphasizing router reliability: mistaken PRM judgments become more consequential on harder problems.

3. Narrative, Core Assumptions, and Unavoidable Tradeoffs

RSD has four layers.

1. Move from token proposals to reasoning-step proposals: the draft generates until the \n\n separator, making a complete step the acceptance unit.
2. Replace likelihood matching with PRM scoring: high-scoring draft steps are retained; low-scoring ones are discarded and regenerated by the target from the same context.
3. Formalize acceptance probability as a mixed policy: output is a context-dependent mixture of draft and target, no longer claimed to equal the target distribution.
4. Choose thresholds under a cost budget: if the draft can handle only a specified share of steps, a binary top-quantile reward threshold is theoretically optimal. Practice uses fixed δ=0.7and reports task-tuned δ* as an upper bound.

The argument assumes that the target's expected reward is at least the draft's at every context (Equation 1, PDF p.2). Yet Appendix Table B.1 contains counterexamples: on CN Middle School, the draft 75.2、target 72.3; on College Math, the draft 48.0、target 46.8(PDF p.15). The paper explains that RSD can exploit draft advantages, reaching 78.2 and 48.2. But the theorem's sufficient condition therefore fails on some tested subdomains and cannot unconditionally cover the full evaluation distribution.

The tradeoff is clear: standard SD preserves the target distribution but accepts only token-compatible proposals; RSD accepts apparently correct semantic steps and may improve accuracy, but depends on PRM bias and loses distributional fidelity. Final-answer accuracy often matters more than exact sampling in mathematics. For calibrated probabilities, copyright filters, safety policies, or auditable sampling, the tradeoff may be unacceptable.

4. Equations and Actual Execution

Given question x, split reasoning into steps y=(y_1,…,y_T), with the context at step i written as z_i=[x,y_1,…,y_{i-1}]. Draft and target distributions are P_m(y_i|z_i) with P_M(y_i|z_i), and process reward is r(y_i,z_i)。

The paper assumes (Equation 1, PDF p.2):

E_{y~P_M(·|z)}[r(y,z)] ≥ E_{y~P_m(·|z)}[r(y,z)]。

Defining acceptance function ω_r(y|z)∈[0,1] , RSD's step distribution is (Equation 2, PDF p.3):

P_RSD(y|z)=ω_r(y|z)P_m(y|z)+ν(z)P_M(y|z)，

where ν(z)=1−E_{y~P_m(·|z)}[ω_r(y|z)]. The first term is accepted draft probability mass; the second restores mass through target sampling after rejection. This exposes the central difference from standard SD: there is no residual correction restoring P_M, so P_RSD≠P_M generally holds.

If the draft's average acceptance rate is constrained to γ, Proposition 2.3 gives the optimal binary reward-ranked policy (Equation 3, PDF p.4):

ω_r^*(y|z)=1[r(y,z)≥δ_γ(z)]，

δ_γ(z) is the draft reward distribution's (1−γ) quantile. Implementation does not estimate a quantile online for every context; it uses global threshold 0.7. The theorem explains selection when the threshold is known, not why one global 0.7 is optimal for all tasks.

Input: question x, draft Pm, target PM, process reward model PRM, threshold δ
Context z ← x
while no termination marker:
    draft_step ← Pm.generate(z, stop="\\n\\n")
    score ← last-step score from PRM(z + draft_step)
    if score ≥ δ:
        Accept draft_step
    else:
        Discard draft_step
        target_step ← PM.generate(z, stop="\\n\\n")
        Accept target_step
    z ← z + accepted step
return answer extracted from final text

The official main_online.py execution path is heavier than this pseudocode: draft, target, and PRM run as three independent OpenAI-compatible servers/clients. Each step first requests the draft, then tokenizes the entire accumulated prompt+response and sends it to the PRM embedding endpoint; rejection triggers a subsequent target request. The README explicitly requires at least 3 GPUsfor vLLM online mode because the current configuration cannot colocate models on one GPU. This does not invalidate the algorithm, but the FLOPs model omits RPCs, repeated-prefix scoring, KV/memory residency, and cross-service queues.

5. Key mechanisms and engineering details

6. Datasets, benchmarks, and metrics

All pp values are percentage-point differences;relative improvement = (RSD − baseline) / baseline × 100%. Relative percentages do not replace confidence intervals. With no reported variance, changes below 1 pp deserve particular caution.

7. Complete Main-Results Matrix: Six Benchmarks × Seven Methods

The following comes from Table 2 (PDF p.5), using the Qwen2.5-Math family.δ* uses task-optimized thresholds and represents a tuning upper bound; fixed δ=.7 is closer to one-time deployment. Bold indicates the column maximum, but total FLOPs differ, so this is not a strictly budget-matched ranking.

For fixed-threshold RSD with a 7B PRM, column-wise changes versus the 7B target are (Table 2):

The most striking single result is small-PRM GPQA: 38.4 versus the target's 32.8,+5.6 pp / +17.07%; yet the same configuration drops Olympiad from 41.2 to 39.6. Larger PRMs are not uniformly better either. Benefits depend on task-specific interactions between routing and rewards, not a monotonic model-size law.

7.1 72B results and FLOPs

In Table 2's 72B block, the target averages 71.8across six tasks; token SD reaches 71.1. RSD with a 7B draft / 72B target / 7B PRM reaches, at a fixed threshold, 72.7, or 72.9 with optimized thresholds. On MATH-500, RSD scores 88.0versus the target's 85.6, namely +2.4 pp / +2.80%. Figure 4 and the text state that this uses approximately 4.4× fewer FLOPs. This is a 4.4× reduction in the paper's FLOPs proxy, not a 4.4× latency speedup: seconds, tokens/s, concurrency, energy, and P99 are unreported.

7.2 General-instruction boundaries

On AlpacaEval, draft 1.5B, target 7B, and RSD win rates are respectively 7.09、24.47、18.85; RSD has 65% of tokens generated solely by the draft, but trails the target by 5.62 pp / 22.97% relative(PDF p.8, §4.5). This matters more than the average +0.8 pp: without a dedicated general-domain PRM, mathematical success does not directly transfer to open-ended instruction following.

8. Layered ablations, hyperparameter sensitivity, and transitions

8.1 Search strategies: Routing within reasoning versus selecting at sequence endpoints

Table 3 (PDF p.6) compares MATH-500 / GSM8K / Minerva Math: the 1.5B model 73.8/85.0/29.0；Process BoN‑8 75.8/87.8/32.7；BoN‑16 76.0/87.9/31.2；Beam‑4 78.2/88.9/33.5；Beam‑8 78.2/88.4/32.4；RSD 82.6/94.5/34.6. RSD can escalate every step to the target, whereas BoN/beam mainly expands the small model's candidate space. The gap can reflect stronger correction sources, not just greater search width.

8.2 Acceptance functions: Binary gating is best at comparable cost

Figure 6 (PDF p.7) reports MATH-500 accuracy at similar costs for constant, binary, clipping, sigmoidal, and logistic weights as 77.6、84.6、81.8、81.4、79.2. Binary beats constant by 7.0 pp. Continuous weights randomly admit middling steps; binary gating concentrates limited draft acceptance on the PRM's highest-ranked steps. However, this is tested on one main benchmark/model setup without repetitions across PRM calibrations.

8.3 Threshold sensitivity: 0.7 is near a peak, not a universal constant

Appendix Table B.2 (PDF p.16) sweeps fixed thresholds:

0.7 is best on the tested grid, but the peak-to-trough range is only 0.7 pp for 7B and 1.1 pp for 72B. Without seed variance, these may not exceed sampling noise. More importantly, with general Qwen and a fixed 7B PRM, average 59.4 falls below the target's 59.9, while task-optimized thresholds reach 62.1(Table 2). Fixed-threshold transfer remains unresolved.

8.4 Difficulty transitions: Routing shares change continuously with difficulty

Draft-only completion shares for MATH Levels 1→5 are 84%、67%、58%、44%、19%; target-intervention accuracy gains are +4.7、+5.6、+6.7、+16.4、+15.7 pp(Appendix Figure B.1, PDF p.17). Level 3→4 is the sharpest shift: draft-only completion drops another 14 pp while target gains jump 9.7 pp. This supports difficulty-aware routing, but exposes coarse global thresholds; better control should estimate current-step risk × remaining budget.

8.5 PRM scale and model merging

Table 5 (PDF p.8) gives Skywork PRM 1.5B/7B averages of 65.9/66.1, and Qwen Math PRM 7B/72B averages of 65.9/66.5. Scaling PRM from 7B to 72B adds only 0.6 pp while substantially increasing residency and computation. Table 4 (PDF p.7) reduces model count through merging: draft+PRM reaches 65.0, below the separate 1.5B PRM's 65.9; target+PRM reaches 66.7, above the separate 7B PRM's 66.1. Actual wall-clock savings depend on kernels, batches, and memory, and are not measured.

9. Statistical Significance: What Cannot Be Claimed

The paper reports no multiseed means, standard deviations, bootstrap CIs, paired McNemar tests, or significance p-values. Main RSD and SD use mostly deterministic temperature-0 decoding, reducing generation randomness but not finite-benchmark error, PRM checkpoint selection effects, or threshold-selection bias.

One can say the reported fixed-threshold point estimate averages 0.8 pp higher, not that RSD significantly outperforms the target in general. On MATH-500, 84.6% versus 83.2% is approximately 423 versus 416 correct, only 7 problems. Without per-problem predictions and paired tests, the source of the difference is unknown. AlpacaEval's −5.62 pp clearly flags external-validity concerns, but is not itself a formal significance result.

10. Contributions, reusable designs, and transfer potential

RSD's main contribution is decoupling speculation unit、verification signal、fallback model these three elements. Tokens can become reasoning steps, code blocks, tool calls, or action segments; acceptance can use PRMs, type checks, unit tests, constraint solvers, or safety classifiers; fallback can be a large model, retriever, or human approval. This abstraction is more reusable than threshold 0.7.

Second, it formulates the quality–cost tradeoff as acceptance-function optimization. Proposition 2.3 says that with trustworthy PRM rankings and fixed acceptance budgets, high-reward drafts should be retained preferentially rather than uniformly or linearly mixed. In code generation, r can represent compile/test signals; for agents, a step can be action+observation with state-constraint checks; in visual-language reasoning, acceptance can also inspect image evidence.

Third, it reveals a new systems boundary for test-time scaling: fewer large-model calls do not automatically reduce end-to-end latency. Three-model residency, accumulated-prefix PRM scoring, and serial fallback may consume gains. Merging PRM heads into draft/target models, reusing KV, batching step verification, or asynchronously prefetching target work could turn FLOPs savings into service benefits.

11. Related Work: Origins, Gaps, RSD's Response, and Inherited Designs

11.1 Fast Inference via Speculative Decoding（Leviathan et al., ICML 2023）

Problem addressed: a small draft proposes tokens, a large target verifies them in parallel, and a corrected distribution preserves exact target sampling; the paper reports roughly 2–3× speedups without changing the distribution.Remaining gap: token-level agreement rejects semantically correct but differently worded steps in long CoT.RSD's response: enlarge acceptance units to reasoning steps and use PRMs to judge quality.Design transfer: retain draft/verify/fallback structure, replacing unbiased sampling with a reward-guided mixed policy.

11.2 BiLD（Kim et al., NeurIPS 2023）

Problem addressed: call the large model when small-model confidence is low, with rollback correction over recent tokens.Remaining gap: small-model confidence is not reasoning correctness, and token rollback remains local.RSD's response: introduce an independent PRM and use task-related process rewards for whole-step takeover.Design transfer: preserve small/large cascades and on-demand fallback, replacing softmax confidence with an external verifier.

11.3 Let’s Verify Step by Step（Lightman et al., ICLR 2024）

Problem addressed: PRM800K's roughly 800,000 human step labels show process supervision better localizes mathematical errors than outcome supervision.Remaining gap: PRMs are mainly used to rank candidates, not schedule computation in real time.RSD's response: turn the PRM into a router deciding target escalation at every step.Design transfer: step-level scalar rewards directly supply RSD's acceptance function.

11.4 Math-Shepherd（Wang et al., ACL 2024）

Problem addressed: automatically builds stepwise process supervision, reducing annotation cost and supporting verification and PPO.Remaining gap: automatic labels retain final-answer/rollout-estimation biases and do not implement real-time small/large collaboration.RSD's response: consume public PRMs for inference-time routing without retraining them.Design transfer: scalable automatic process labels make stepwise acceptance practically available.

11.5 Large Language Monkeys（Brown et al., 2024）

Problem addressed: studies repeated sampling and verifiers as test-time compute scales, showing that more samples substantially increase problem coverage.Remaining gap: repeated full sequences recompute easy steps, with cost growing linearly in N.RSD's response: concentrate extra computation on low-reward steps, invoking the target locally.Design transfer: transform multiple-sample selection into serial, local, dynamic allocation of test-time compute.

11.6 Nonlinear technical evolution tree

                          ┌─ repeated sampling / Best-of-N ── sequence-level search
Process supervision PRM ──┤
  ├─ PRM800K              └─ Math-Shepherd ── scalable step labels
  │                                      ╲
  │                                       ╲  step rewards as routing
token speculative decoding ── exact target ╲
  │                                          ├─ RSD ── reward-guided mixed policy
  └─ BiLD ── confidence fallback ────────────╱          │
                                                        ├─ GSI: distribution/KL theory
                                                        ├─ SpecReason: measured latency systems
                                                        └─ Lookahead: parallel steps

Three shifts drive this evolution: from training scale to inference-compute allocation; from exact token agreement to semantic step correctness; and from accuracy/FLOPs to end-to-end latency and distributional fidelity. RSD completes the second shift but not the third.

11.7 Cross-paper comparison

12. Limitations acknowledged by the paper

1. Thresholds require task adaptation: Appendix B.1 (PDF pp.15–16) explicitly notes fixed thresholds may be suboptimal across tasks; task-specific tuning of δ* improves results. Unknown future task distributions require calibration data or online budget control in deployment.
2. General domains lack suitable PRMs: §4.5 / Discussion (PDF pp.7–8) notes AlpacaEval uses only an outcome reward model. Dedicated general-domain PRMs and tuning may help, but current 18.85 still trails the target's 24.47.
3. PRMs could be model-specific: Discussion (PDF p.8) acknowledges use of generic public PRMs rather than verifiers trained for particular draft/target errors. This limits routing quality and increases PRM size costs.

13. Independent Critique with Evidence Chains

1. [Systems validity] The official online implementation needs at least 3 GPUs, splits draft, target, and PRM across services, and resubmits accumulated prefixes to the PRM at every step;thereforetherefore Figure 4's 4.4× FLOPs savings do not establish 4.4× better latency, throughput, or cost. TTFT, per-step RPCs, tokens/s, P50/P95/P99, and memory must be measured.

2. [Theoretical applicability] Equation 1 assumes target expected reward is no lower than draft reward, but Appendix Table B.1 has two subdomains where draft scores of 75.2 and>72.3、48.0>46.8 exceed the target;thereforethe proposition's monotonic-improvement conditions therefore do not cover all actual subdistributions. Theory should incorporate context-dependent expert advantages and learned selection.

3. [Hyperparameter-selection bias] With general Qwen and fixed 7B PRM, the average is 59.4 versus target 59.9, while task-specific δ* reaches 62.1;thereforeoptimized thresholds contain a substantial oracle component. Reporting δ* must distinguish this from an independent calibration split.

4. [External validity] The fixed-threshold mathematics average improves by only 0.8 pp, while AlpacaEval falls from target 24.47 to 18.85 (−5.62 pp);thereforebenefits depend strongly on an available, aligned process reward model and cannot establish general LLM reasoning acceleration.

5. [Statistical evidence] All main tables are point estimates without seed variance or paired significance, and MATH-500's +1.4 pp is approximately 7 problems;thereforesmall gains cannot be judged stable. At minimum, release per-problem outputs and run paired bootstrap / McNemar tests.

6. [Comparison fairness] Majority Vote/BoN sample counts and RSD target calls are not matched on wall-clock time, energy, or GPU budgets;thereforeTable 2 ranks preset configurations rather than identifying the best method under equal service budgets. Separate comparisons should fix FLOPs, memory, and P99.

7. [Format robustness] Implementation uses \n\n as the reasoning-step boundary;thereforeshort-line CoT, JSON, code, tool calls, or different chat templates change routing granularity. Compare learned segmentation, token-budget chunks, and semantic boundary detection.

8. [Safety and distribution] RSD deliberately outputs a reward-guided mixture rather than the target distribution, and PRMs may overreward high-scoring patterns;thereforethis creates reward-hacking risks for safety policies, calibrated probabilities, and sensitive generation. Report KL, rejection reasons, OOD PRM calibration, and adversarial step tests.

14. Actionable Open Challenges as Head-to-Head Experiments

14.1 Theory

14.2 Engineering applications

14.3 Emerging directions

15. Recent-Work Radar (Verified Links)

1. Guided Speculative Inference（GSI, 2025）：arXiv:2506.04118. Directly addresses RSD's distribution-fidelity gap through reward-tilted policies and KL bounds; compare at equal reward and latency budgets.
2. SPECS（2025）：arXiv:2506.15733. Uses small-model reasoning drafts and dynamic switching to reduce test-time scaling costs. It is closer to service decisions, but threshold transfer needs independent verification.
3. Scaling Speculative Decoding with Lookahead Reasoning（2025）：arXiv:2506.19830. Combines token speculation with step-level lookahead, reporting peak SD speedup increasing from 1.4× to 2.1×; extends serial step routing into parallel future steps.
4. SpecReason（NeurIPS 2025）：arXiv:2504.07891. Its public abstract reports 1.5–2.5× speedup over vanilla LRMs and +1.0–9.9% accuracy, with measured latency. It is a key comparison for whether semantic speculation delivers systems gains.
5. Speculative Streaming（EMNLP 2025 Main）：ACL Anthology. Uses multistream attention to verify existing tokens and predict future ones in one forward pass, offering a route without a separate resident draft server.
6. SpecGuard / From Tokens to Steps（2026）：Recent ACL Anthology search entry. Raises speculative acceptance from tokens to steps with quality safeguards; check whether it supplies the P99 and false-acceptance breakdown missing from RSD.
7. HeteroSpec（2026）：Recent ACL Anthology search entry. Studies heterogeneous small/large-model collaboration; cross-device scheduling would directly address RSD's three-service residency bottleneck.
8. Calibrated Speculative Decoding（2026）：Recent ACL Anthology search entry. Focuses on calibrated acceptance rather than a single optimal threshold. Comparison with RSD's δ=.7 should jointly report ECE, call-rate error, and OOD performance.

16. Conclusion

RSD offers a transferable design:treat small-model outputs as work an external verifier can accept, rather than guesses that must match a large model token by token. On Qwen2.5-Math-7B, fixed thresholds raise the six-task average from 65.3 to 66.1; on 72B MATH-500, from 85.6 to 88.0 with 4.4× fewer proxy FLOPs. The reusable contribution is step-level routing, not the constant 0.7.

Evidence remains algorithmic: three-model services, full-prefix PRM scoring, absent wall-clock/P99 measurements, and no multiseed statistics leave systems efficiency unresolved. The next priority is comparing target-only, token SD, GSI, and SpecReason at matched accuracy, hardware, and concurrency on P99, energy, and memory, alongside OOD PRM calibration. Passing that test could turn a clever reasoning algorithm into a reliable online serving strategy.

Footnote: Numbers and index status. All experimental values cite paper tables, figures, or PDF pages. Unreported variance, latency, throughput, P99, energy, and GPU-hours remain explicitly missing rather than fabricated.ml-report-index.md The core paper, domain, venue, and one-sentence conclusion were added to the index in this round.