ReActR

4.1 Data units

Let the problem be $x$, the student's initial trajectory be $y^{(0)}$, and the answer checker be $V(x,y)\in\{0,1\}$. For an incorrect trajectory, the teacher generates a structured reflection:

$$z=(c,\ell,e,y^{(1)},g),$$

where $c$ denotes one of six error labels,$\ell$ the error location,$e$ an explanation of error propagation,$y^{(1)}$ the corrected trajectory, and$g$ a transferable lesson. A training example can be written as $(x,y^{(0)},z)$. Correct trajectories also retain a reflection turn confirming correctness, preventing the model from learning to overturn every answer whenever it reflects.

The supervised stage optimizes standard autoregressive negative log-likelihood:

$$\mathcal L_{\text{SFT}}(\theta)=-\sum_{(x,s)\in\mathcal D}\sum_t \log \pi_\theta(s_t\mid x,s_{<t}),$$

where $s$ concatenates initial reasoning, reflection, correction, and summary. The paper introduces no new SFT equation; the expression above is simply a standard formalization of the described training process.

4.2 Five reward states

Write $a_0=V(x,y^{(0)})$, the reflection judgment as $d\in\{\text{correct},\text{wrong}\}$, and the corrected answer as $a_1=V(x,y^{(1)})$. Figure 6 gives five cases:

This can be written as:

$$r=\mathbb 1[a_0=1,d=\text{correct}] +0.3\,\mathbb 1[a_0=0,d=\text{wrong}] +0.7\,\mathbb 1[a_0=0,d=\text{wrong},a_1=1].$$

This definition reveals an easily overlooked property: correctly confirming a correct answer and fully repairing an incorrect one both earn 1.0. The reward therefore encouragestask success, rather than more reflection for its own sake. However, it does not distinguish confident correctness from lucky correctness or score the causal faithfulness of the reflection text.

4.3 Within-group comparison in GRPO

The paper specifies 5 responses sampled per problem, 3 training epochs, a fixed learning rate of $10^{-6}$, and verl to run GRPO (Appendix A.2, PDF p.11). It does not provide the complete objective. Standard GRPO can formalize the description as follows: for $G=5$ responses to the same problem, first compute group-normalized advantages,

$$\widehat{A}_i=\frac{r_i-\mu_r}{\sigma_r+\epsilon},$$

then use a clipped ratio, $\rho_i=\pi_\theta(y_i\mid x)/\pi_{\theta_{old}}(y_i\mid x)$ Optimization

$$J(\theta)=\frac{1}{G}\sum_i \min\left( \rho_i\widehat{A}_i, \mathrm{clip}(\rho_i,1-\varepsilon,1+\varepsilon)\widehat{A}_i\right) -\beta\,\mathrm{KL}(\pi_\theta,\pi_{ref}).$$

This isa standard restatement of the procedure, not an equation explicitly presented in the ReActR paper. The paper does not report $\varepsilon$、$\beta$, KL implementation details, or their sensitivity, so this is insufficient to reproduce the full GRPO configuration.

4.4 Pseudocode

Input: MATH and GSM8K problems; Qwen3-4B student; Gemini 2.5 Pro teacher
1  Sample initial student reasoning at temperature=0.6, retaining actual errors
2  Use the teacher to synthesize difficult QM/CC/CA errors on MATH levels 3–5
3  For each error: label type → locate the first critical error → analyze propagation
                  → generate a corrected trajectory → summarize a lesson
4  Mix 25K error–correction examples with 25K correct trajectories
5  Perform 4 epochs of LoRA SFT
6  Sample 5 responses per problem and compute group advantages using five-state rewards
7  Train with GRPO for 3 epochs
Output: a reasoning model that confirms correct answers, detects errors, and corrects them

7.1 Qwen3-4B

Source: core paper, Table 2 (PDF p.7 / proceedings p.43023). Relative to the strongest comparison, LEMMA, ReActR improves the five individual benchmarks by 4.5、5.2、6.8、5.3、2.9 pp, and Overall by 4.9 pp / 9.35%. Relative to ordinary SFT, Overall improves by 14.9 pp / 35.14%。

7.2 Mistral-7B

Source: Table 2 (PDF p.7). Relative to LEMMA, the five individual improvements are 3.9、4.8、3.3、5.8、0.3 pp, and Overall by 3.6 pp / 12.08%. Relative to SFT, Overall improves by 13.6 pp / 68.69%. The large relative percentage partly reflects Mistral's low SFT baseline of 19.8; 68.69% should not be read as an almost twofold increase in absolute capability.

7.3 Llama3-8B

Source: Table 2 (PDF p.7). Relative to LEMMA, the five individual improvements are 4.7、2.8、4.4、4.1、1.2 pp, and Overall by 3.5 pp / 9.80%; relative to SFT, Overall improves by 13.5 pp / 52.53%。

7.4 Conclusions across backbones

ReActR exceeds LEMMA in all 15 model × individual-benchmark cells across three backbones, stronger evidence of consistency than a single average. Yet this is not statistical significance: shared training data and teacher signals could bias all three backbones in the same direction, and the paper lacks multiple random seeds. Quantitatively, ReActR's Overall advantage over LEMMA is 3.5–4.9 pp, much smaller than its 13.5–14.9 pp advantage over ordinary SFT. This suggests that a substantial share of the gains comes from the broader category of better mathematical training data, rather than solely from ReActR's distinctive staged rewards.

8.1 Data design and RL ablations

The paper provides the following table for Qwen3-4B (Table 3, PDF p.7 / proceedings p.43023):

Three nonlinear observations are more informative than the routine conclusion that every component helps. First,Out-Both scores 19.4 pp below the full model on MATH, a larger loss than the sum of separately removing error type (9.5) and difficulty (5.4), 14.9 pp. The two curriculum signals may reinforce each other, but without multiple seeds, the extra 4.5 pp cannot be established as a stable interaction. Second, All-Error fails outright: valuable errors do not imply that more errors are always better; 25K correct trajectories anchor correct policies. Third, removing GRPO still yields 88.4 / 62.1, showing that most gains are established during SFT. GRPO adds 2.7 / 3.6 pp rather than accounting for the entire improvement.

8.2 Data scaling curves

Figure 7a (PDF p.7) shows GSM8K and MATH generally improving as augmented data grows from 10K to 50K, with a stronger rise on MATH. The plot lacks per-point numerical labels, so this report does not fabricate an exact matrix; it supports only the conclusion that clear saturation has not occurred in the tested range. Figure 7c also shows average performance increasing with more MetaMath data, again as a plot without an exact table. This supports complementing the error curriculum with high-quality correct trajectories, but does not permit calculating marginal gains per 1K examples.

8.3 Changes in error distribution

Figure 7b shows the proportions of QM and CA falling after augmentation, while CC and Others increase. The authors attribute this to targeted synthesis adding more complex conceptual errors. Without exact numerical labels on the bars, this report retains only the directional conclusion. Data curation actively changes the error distribution rather than simply replicating natural frequencies; the resulting distribution may reflect Gemini 2.5 Pro's pedagogical imagination more than real user mistakes.

8.4 Hyperparameter sensitivity

Not reported in the paper. The reward split is fixed at 0.3 for detection + 0.7 for correction, with no comparison against 0.1/0.9, 0.5/0.5, or continuous scores. GRPO group size is fixed at 5, with no scaling curve. The SFT learning rate $10^{-5}$, GRPO learning rate $10^{-6}$, 4 / 3 epochs, and temperature 0.6 also lack sensitivity analyses. It is therefore unknown whether the method is fragile to reward ratios, sampling temperature, or training duration, or whether 0.3/0.7 is mechanistically justified or simply a tuned setting.

8.5 Assessing phase transitions

There are two candidate phase transitions, with different evidence strengths.All-Error failure shows training collapse when the mixture moves from having correct anchors to all errors: a qualitative transition. However, the authors do not sweep error proportions of 10%, 25%, 50%, 75%, and 100%, so the threshold is unknown. The 10K→50K data curve shows continuing gains without an abrupt change, offering no basis for a scaling phase-transition claim.

10.1 Actual contributions

First, errors become curriculum units. Many earlier methods either repeatedly self-evaluate at inference time or use wrong answers as negative training examples. ReActR connects error type, difficulty, and propagation analysis into a curation loop, making errors control variables for training coverage rather than merely negative labels.

Second, rewards separate diagnosis from repair. Successful error detection earns 0.3 even when correction fails, providing a learning signal for partially correct internal decisions. Though coarse, this reward better matches reflection's staged structure than an all-or-nothing final-answer reward.

Third, mixing positive and negative trajectories becomes necessary. All-Error fails and All-Correct substantially underperforms. The reusable design is to stabilize the basic policy with correct trajectories and develop recovery with incorrect ones, not simply to feed the model errors. This lesson is more transferable than any particular error label.

10.2 Reusable modules

1. Error ontology + quota sampling. For code, categories could include syntax, types, boundaries, algorithms, complexity, and tool calls; for agents, omitted observations, conflicting plans, tool selection, state updates, and stopping errors.
2. Locating the first error. Identify the earliest decision that causes subsequent divergence before rewriting everything; this can shorten the credit-assignment horizon.
3. Reflection lessons. Produce a retrievable rule after each correction to support episodic memory or an error notebook.
4. Staged verifiable rewards. Where unit tests, theorem checkers, simulator states, or database constraints exist, reward detection, repair, and final completion separately.
5. Mixed correct/incorrect curricula. Retain anchors for correct policies while gradually increasing difficult errors, avoiding a policy that only learns to doubt.

10.3 Transfer potential

Code is the most direct transfer target because unit tests can check both initial and corrected versions, though test deletion and hard-coding must be prevented. For tool agents, error propagation can describe how a wrong tool call contaminates later state, with environment rollback verifying repairs. Scientific QA is harder because final answers may not be automatically checkable; it requires citation consistency, unit checks, symbolic constraints, or external simulators rather than direct mathematical answer matching. Open-ended writing is the weakest fit because correction lacks an objective verifier, allowing teacher preferences to dominate rewards.

11.1 Self-Refine: from inference-time self-feedback to trainable reflection

Problem solved. Self-Refine uses the same model to generate, provide feedback, and iteratively revise without additional training. It reports approximately 20% average absolute improvement across 7 task types (NeurIPS 2023).
Remaining gap. It operates mainly at inference time, without training-level guarantees of reliable error detection, appropriate stopping, or feedback identifying the first critical error.
ReActR's response. ReActR moves generate–feedback–correct loops into post-training data and uses an answer verifier to reward error discovery and final correction.
Design transfer. Self-Refine's iterative loop becomes ReActR's multiturn sample structure, augmented with an error ontology, mixed curriculum, and RL signal.

11.2 RefAug: learning reflective explanations beyond the answer

Problem solved. RefAug adds reflection and extended information to instruction data, teaching explanations and alternative approaches beyond answer imitation.
Remaining gap. General reflection augmentation does not explicitly control error type and difficulty or split detection and correction into separate rewards.
ReActR's response. ReActR targets difficult errors using six categories and MATH levels 3–5, and includes RefAug as a 30K baseline. Qwen Overall is 47.6 versus 57.3, a 9.7 pp gap (Table 2).
Design transfer. The idea of learning an additional layer of metadata after the answer is retained, but general reflection becomes first-error localization, propagation analysis, and lessons.

11.3 LEMMA: from error-correction data to a type–difficulty curriculum

Problem solved. LEMMA systematically trains mathematical reasoning on incorrect solutions and corrections, making it ReActR's most direct technical predecessor.
Remaining gap. Without curation by type and difficulty, error examples can be dominated by frequent, shallow mistakes; supervised correction alone also cannot separately reward detection.
ReActR's response. ReActR adds error categorization and difficulty guidance, followed by GRPO. Overall exceeds LEMMA on all three backbones: Qwen +4.9, Mistral +3.6, and Llama +3.5 pp (Table 2).
Design transfer. LEMMA's error–correction trajectories provide the data skeleton; ReActR adds curriculum control and staged rewards. LEMMA uses 60K examples versus ReActR's 50K, suggesting some sample efficiency, though this is not a strictly compute-matched experiment.

11.4 DeepSeek-R1: from verifiable outcome rewards to reflection-specific rewards

Problem solved. DeepSeek-R1 shows that large-scale RL can induce long reasoning chains and self-checking from verifiable mathematical and coding rewards.
Remaining gap. Sparse final-correctness rewards cannot readily distinguish detecting an error without fixing it from failing to detect it at all, creating a long credit-assignment horizon.
ReActR's response. ReActR explicitly assigns 0.3 to error detection, 0.7 to final correction, and 1.0 to correctly confirming a correct answer.
Design transfer. The verifiable-outcome RL framework remains, but its reward structure is redesigned around a reflection state machine. Dependence on gold answers limits open-domain transfer.

11.5 Nonlinear technical evolution tree

                         ┌─ Self-Refine: inference-time generation→feedback→revision
                         │          └─ Multiturn reflection format ──────┐
Correct-trajectory SFT ──┼─ RefAug: metadata beyond answers ─────────────┤
                         │                                               ├─ ReActR
Error-trajectory learning┼─ LEMMA: error→correction supervision ──────────┤   ├─ Six error types
                         │          └─ Correction-data skeleton ─────────┤   ├─ Type×difficulty curriculum
Verifiable-outcome RL ───┴─ DeepSeek-R1: final-answer reward ─────────────┘   └─ 0.3 detection+0.7 correction

The technical turning point is the convergence of three strands: inference-time self-revision supplies the interaction format, error-correction data supplies supervision, and verifiable RL supplies optimization. ReActR connects curation and reward design through error types and difficulty.

11.6 Cross-paper comparison

14.1 Theory

Why do incorrect trajectories add information? Run a strict head-to-head with fixed problems, token counts, teacher, and final correct solutions. Divide 50K examples into (A) repeated correct solutions, (B) random errors + correct solutions, (C) actual errors + correct solutions, and (D) type-quota errors + correct solutions. Compare sample complexity and error-detection mutual information to separate decision-boundary information from simply longer text.

Is the reward decomposition identifiable? Fix the SFT checkpoint and compare five rewards: final-only, 0.1/0.9, 0.3/0.7, 0.5/0.5, and a learned critic. Besides final accuracy, measure error-detection AUPRC, conditional correction rate $P(a_1=1\mid a_0=0,d=wrong)$ , and unnecessary answer-reversal rate. This is needed to establish whether 0.3 has mechanistic significance.

Collapse threshold of the correct/incorrect mixture. Sweep error proportions from 0% to 100% in 10% increments, with 3 seeds each. Plot capability, calibration, and training stability to locate the critical region before All-Error failure rather than comparing only endpoints.

14.2 Engineering and applications

Compute-matched comparisons. Give ReActR, LEMMA, RefAug, and correct-trajectory SFT the same 50K examples, total tokens, LoRA rank, and GPU hours. Report training cost and Pass@1 together. Current differences in sample counts confound sample efficiency with method gains.

Service-level value. At the same accuracy target, compare no reflection, one fixed reflection, confidence-triggered reflection, and ReActR on mean output tokens, P50/P95 latency, energy, and cost. Test whether activation only when errors are likely can offset multiturn overhead.

Transfer to code. On reproducible subsets of HumanEval+, MBPP+, and SWE-bench Verified, compare final-only RL, a unit-test error taxonomy, and ReActR-style first-error localization. Rewards must prevent test tampering. Determine whether gains concern compilation errors, logic errors, or repository navigation.

14.3 Emerging directions

Rollback-enabled reflection for tool agents. Record the first incorrect tool call in WebArena or τ-bench, and compare full restart, rollback to the first error, and textual reflection without rollback. Measure task success, additional calls, and depth of state contamination to test whether propagation analysis actually shortens recovery.

Joint training with process verifiers. Compare ReActR head-to-head with an independent process reward model: under equal rollout counts, test final-answer rewards, step rewards, and their combination. Use a human-labeled subset to determine whether the verifier merely reproduces teacher bias.

Iterative correction of compound errors. Construct a diagnostic set with two known errors per problem, comparing one-round dominant-error repair against up to three rounds of sequential correction. Measure first-error localization, exposure of the second error, cumulative success, and token cost, directly addressing the authors' limitation.