RL for Long-Horizon Tool Agents: A Five-Axis Dissection of Agent-STAR and an Underappreciated Reverse Trend

Before reading: disclosure of limitations

1. Venue attribution: The Comments field on the core paper's arXiv page only lists the code URL and does not claim acceptance at any conference. Its HTML formatting appears to use the ICLR conference template, but on OpenReview I could not find a corresponding submission page, so I classify it as a high-quality arXiv preprint. Reason for relaxing the venue requirement: the paper fully releases its data, SFT trajectories, and six checkpoints, making it substantially more reproducible than contemporary work on the same topic.
2. Deduplication index unavailable: This was a scheduled cloud task, and the user's desktop was not connected; ml-report-index.md could not be accessed, so the 14-day duplicate check could not be performed. The candidate pool contained 25 papers; shuf -i 1-25 -n 3 → [3, 17, 15]; selected #3. Redraws: 0.
3. Source boundaries: All performance numbers come from Tables 1–4 and Appendix Tables 8–9 of the arXiv HTML full text, v1. Figures 5/6/7/8 are raster images; values in those figures, such as exact success rates at 5%/10% tool failure and the 500-prompt setting, could not be extracted. Each affected location is explicitly marked below.
4. Statistical significance: I I found no standard deviations, confidence intervals, or multi-seed description anywhere in the full text. All percentage-point differences below are differences between point estimates from single runs.

All links below were actually visited and confirmed accessible (2026-08-06). Items that could not be verified are explicitly marked “not found.”

2.1 Motivation: A 0.6% That Has Remained Unsolved for Two Years

The paper targets not a vague claim that “agents are hard to train,” but a strikingly concrete set of numbers:

• On TravelPlanner, GPT-4's final pass rate is 0.6%(original TravelPlanner paper, ICML 2024)
• Two years later, even top-tier models such as Kimi-K2.5 still achieve below 15% success(STAR introduction)
• STAR's own starting point: Qwen2.5-Instruct 1.5B 0.0%, 3B 0.0%, 7B 0.1%(Table 1, Base row)

Why it is worth solving, ranked by persuasiveness

1. It is a clean slice of “long-horizon” behavior. Each trajectory requires an average of 9.2 tool calls and 10.3K tokens(Appendix Table 8). By contrast, Search-R1-style tasks typically converge after 1–3 retrievals. A length of 9.2 means that any step-level error can be amplified by later steps.
2. Its reward is naturally decomposable. The evaluation protocol separates constraints into commonsense and hard-constraint dimensions, each with micro (fraction) and macro (0/1) levels—a ready-made four-level ladder of reward density, requiring no manually designed process reward model. This makes STAR's density ablations possible.
3. It is an extreme case of sparse rewards. A base success rate of 0.0% means that with a success-only reward, all initial gradients are zero.

Novelty relative to predecessors: From One Axis to Five

The common weakness of predecessors is single-axis optimization: Search-R1 adjusts rewards, DAPO adjusts algorithms, and Planner-R1 adjusts reward shaping. A complete 4×3×4×3×3 grid is infeasible, so STAR ablates one axis at a time while fixing the others. This is both what makes the study feasible and its greatest methodological weakness (see Critique ⑥ in 2.6).

2.2 The Paper's Narrative: A Chain of Elimination

STAR's argument is not “we propose X → X works,” but a chain of elimination forced by the evidence:

1. Why not apply RL directly? Base success is 0.0% → all samples in each GRPO group receive the same reward → the advantage-normalization denominator is 0 → no gradient. An SFT warm start is necessary. This step is forced, rather than a design preference.
2. Where does the SFT data come from? Manually annotating 9.2-step trajectories is too expensive → run ReAct using DeepSeek-V3.2-Exp-Thinking → rejection-sample only successful trajectories → 1,198 retained from 5K queries (24.0% retention). That 24% is itself a signal: even a top-tier teacher fails on TravelPlanner 76% of the time.
3. How dense should the reward be? Base success is 0 → dense rewards are needed → but dense rewards may overfit the task-specific scoring function → design three density levels, Sum / Macro / Success, plus a Curriculum transition. The implicit tradeoff is stated explicitly: exchange in-domain convergence speed for OOD generalization.
4. What counterintuitive result appears? Curriculum is best at both 1.5B (34.9) and 3B (49.9), but at 7B it falls to 57.0, trailing Sum's 62.8 by 5.8 pp. This overturns the default assumption that “smarter reward scheduling is always better.” The authors infer that model capacity itself supplies exploration ability; once capacity is sufficient, external exploration assistance becomes noise.
5. Does that inference generalize? Table 4 checks the algorithm axis: at 1.5B, ARPO (37.5) > DAPO(36.9) > GRPO (30.1), while at 7B the ordering reverses completely, GRPO (62.8) > DAPO(58.4) ≈ ARPO(58.3). Two independent axes support the same conclusion—the strongest part of the paper's argument.

2.3 Dissecting the methodology and mechanisms

Formal definition: Evaluation components directly serve as reward primitives

GRPO objective (KL-free variant)

STAR's Three-Stage Pipeline

Key Mechanisms

3.1 Main Results Matrix (Table 1, TravelPlanner Test Set of 1,000 Cases, Success %)

3.2 Complete Table 1 (Six Metrics × Three Sizes × Six Methods)

Three Nontrivial Observations

1. CS Micro is already saturated, but Success is far from saturated. 7B-Sum achieves 96.9% CS Micro but only 62.8% Success. The bottleneck is entirely at the macro level of hard constraints(66.5%): the remaining errors are not “commonsense mistakes,” but failures to satisfy the final budget / preference constraint.
2. The transition lies between 3B and 7B. From 3B→7B, Curriculum improves by only 7.1 pp (49.9→57.0), while Sum improves by 15.8 pp (47.0→62.8)—an 8.7 pp gap in gains, the strongest directional reversal in the table.
3. RL is the main contributor in-domain. 7B: SFT 19.7 → RL 62.8. RL contributes +43.1 pp, 2.2 times the SFT contribution (19.6 pp). The OOD data in the next section sharply challenges this conclusion.

3.3 OOD Results Matrix (Table 2, Seven Knowledge-Intensive QA Benchmarks, %)

The tool environment is replaced with local Wikipedia retrieval. This transfer setting has the same tool form but entirely different task semantics, testing whether “tool orchestration ability,” rather than “travel knowledge,” transfers.

🔴 The Paper's Most Persuasive Quantitative Evidence (Derived from Table 2, Not Explicitly Stated)

List the “net OOD contribution of RL relative to SFT” by model size alongside the in-domain contribution at the same size:

These diverging trends imply that as models grow, RL increasingly resembles “task-specific overfitting” rather than teaching transferable tool orchestration ability. Extrapolating beyond 7B would put the net OOD contribution below 0. Indeed, at 7B, choosing Curriculum, recommended for 1.5B/3B, or Sum, the strongest in-domain reward, already produces negative OOD returns:

• 7B Sum: 36.7 vs SFT 41.9 = −5.2pp(RL makes the model worse on OOD than SFT alone)
• 7B Curriculum: 41.1 vs SFT 41.9 = −0.8pp

The paper's Takeaway 2 merely says “overly dense rewards have an alignment tax,” but the data support a stronger proposition: at 7B, two of the four rewards give RL a negative net OOD return; the other two give gains no greater than +1.0 pp. In other words, STAR's OOD competitiveness at 7B comes almost entirely from DeepSeek-V3.2 distillation during SFT, rather than RL.

Supporting evidence: 7B-SFT's OOD Avg (41.9) already exceeds Search-R1 (38.5), trained natively on NQ+HotpotQA, by 3.4 pp. A model that has never seen QA data and was distilled only on travel-planning trajectories beats an in-domain-trained specialized baseline on OOD—a victory for distillation, not RL.

3.4 Data Difficulty Composition Ablation (Table 3, 3B, Curriculum, 1K Prompts)

3.5 RL Algorithm Ablation and Cost (Table 4)

Practical implication: Entropy-driven adaptive rollout methods such as ARPO gain by “compensating for insufficient exploration in small models,” rather than by “improving credit assignment itself.” The original ARPO paper (ICLR 2026 Poster) claims better results on 13 benchmarks with half the tool budget, but its experiments mainly use models at or below 7B—STAR's Table 4 effectively draws a boundary on ARPO's applicability, precisely around the sizes used in ARPO's own experiments. This is an interesting case of papers corroborating and constraining one another.

3.6 Data-Scale Sweep and Environment Stability

Environment stability (Figure 6, 3B only)

The paper states that at ≤5% tool failure, “test performance remains relatively stable, with only minor fluctuations”; at 10%, “final test success drops substantially, with all metrics declining together.” The training curves (Figure 6a) show slower convergence and greater variance.

⚠ I could not read the exact values in Figure 6(there is no corresponding table). The quantitative strength of Takeaway 7 cannot be verified. This is also a presentation weakness of the paper: it is the only one of the five axes with a figure but no table.

4.1 The Real Contribution

It is not “we trained a TravelPlanner state of the art” (62.8% will soon be surpassed), but establishing the proposition that “agentic RL design choices are a function of model capacity,” with empirical evidence in the same direction along two independent axes: reward density and algorithm selection.

This proposition changes how papers should be read: whenever an agentic RL paper claims “method X beats Y,” the reader's first question should be “at what scale?” STAR elevates a hidden, rarely reported experimental dimension—model size—to first-class status.

Reusable Designs Independent of This Paper, Ranked by Transferability

1. The construction of a four-level reward-density ladder(micro/macro × multiple dimensions). Any task whose evaluation protocol decomposes into multiple binary checks can directly adopt it: micro = pass fraction, macro = indicator of passing all checks, combined for the desired density. It depends on no travel-planning-specific property. This is the paper's most reusable contribution.
2. The “exclude from loss / retain for normalization” treatment of overlong trajectories. Applicable to any combination of group-based RL and long trajectories.
3. Rejection-sampling retention rate as a diagnostic of task difficulty. The 24.0% rate signals in advance that the task is difficult even for a strong teacher and can quantify whether an SFT warm start is needed.
4. The “algorithm gain vs. size” comparison-table format(Table 4). Reporting GPU hours alongside performance is a rare act of transparency in agentic RL papers and should become routine.

Implications for Future Work

• The hypothesis that “capacity substitutes for exploration assistance” deserves independent testing. If it holds, the entire research direction of designing smarter exploration mechanisms for LLM agents has a finite value window that closes as base models improve.
• The decline in net OOD gains with size opens a new question: does agentic RL “teach capabilities” or “adjust alignment”? If the latter, all current work marketed through in-domain gains needs an OOD control.

4.2 Limitations · Category I: Author-Reported (Four Points in the Limitations Section)

1. Simulated environment: “While TravelPlanner is a challenging testbed, it remains simulated.”
2. Limited OOD evaluation: “Our OOD evaluation is currently restricted to the knowledge-intensive QA task. This limits our conclusions to single-domain generalization.”
3. Unknown behavior of larger models: “Due to computational constraints, our study focuses on models up to 7B parameters.”
4. Task-specific decomposition and single-factor analysis: “we primarily isolated individual axes, leaving the complex, simultaneous interactions among all dimensions unexplored. Furthermore, our reward shaping relies on task-specific, trajectory-level formulations.”

4.3 Limitations · Category II: Additional Critique (8 Independent Observations)

① No statistical significance reporting, while several takeaways depend on <1 pp differences Experimental design issues

No std / CI / seed count appears anywhere. All evidence for Takeaway 4 (the 1K sweet spot) is the 1K→2K in-domain change of +0.9pp; by comparison, the effect size for Takeaway 1 is 5.8pp. Using the same variance-free experiments to support two conclusions whose effect sizes differ by 6.4× necessarily implies different confidence levels. Yet the paper presents all seven takeaways on an equal footing without distinguishing confidence.

② The key negative result—“RL contributes only +1.0 pp net OOD at 7B”—is sidestepped Gaps in the argument

Data chain: 7B SFT OOD 41.9 → best RL OOD 42.9 (Macro) = +1.0 pp; 7B Sum is 36.7 = −5.2pp. Takeaway 2 says “overly dense rewards impose an alignment tax,” a weaker formulation that assigns responsibility to a single reward design. The actual data support a stronger proposition: at 7B, the expected OOD contribution of the RL stage is near zero, and half the configurations are negative. The positive phrasing “Macro offers the best balance” obscures this fact.

③ No head-to-head comparison with the most direct competitor, Planner-R1 Missing comparison with a key competitor

STAR cites Planner-R1 in related work, and it does the same thing on the same benchmark, reporting 56.9% final-pass rate with only 180 training queries and an 8B model. STAR's best 7B result is 62.8%, at the cost of 1,198 teacher-distilled SFT trajectories + 1K RL queries ≈ 2,198 examples——12.2× Planner-R1's sample count for only +5.9 pp. In STAR's Tables 1 and 4, not a single row is Planner-R1; the claim of “significantly outperforming leading LLMs” compares against general LLMs such as Kimi-K2.5, rather than comparable specialized methods.

Disclosure: Planner-R1 uses 8B while STAR uses 7B, and their SFT setups differ, so these are not strictly identical experimental conditions. Precisely for that reason, a controlled head-to-head is needed instead of separate reported numbers.

④ “Curriculum is best at 1.5B” conflicts with “Curriculum has the worst OOD at 1.5B” within the recipe Gaps in the argument

Table 1: Curriculum's Success at 1.5B is 34.9 (first place). Table 2: the same 1.5B-Curriculum has OOD Avg of 16.9, last among all four rewards, below first-place Macro (25.6) by 8.7pp, and even 7.4 pp below Sum (24.3). Takeaway 1 recommends “staged rewards for small models,” while Takeaway 2 recommends “avoiding overly dense rewards to preserve OOD”—these recommendations directly conflict at 1.5B, and the paper gives no criterion for choosing between them. A practitioner reading only the takeaways rather than Table 2 would choose Curriculum for 1.5B and obtain the worst generalization in the table.

⑤ KL-free training and “overly dense rewards” are not disentangled Experimental design issues

STAR's GRPO variant makes two changes simultaneously—removing KL and using clip-high—both of which amplify policy drift. Sum's OOD degradation at 7B (−5.2 pp vs. SFT) is attributed to reward density, but there is no KL-on / KL-off comparison. Without it, “reward density causes the alignment tax” is a causal claim that has not been isolated and tested.

⑥ One-axis-at-a-time ablations use conditions inconsistent with the recommended configuration Experimental design issues

The paper acknowledges single-factor analysis but underestimates its cost: Table 3 (data difficulty) and Appendix Table 9 (data scale) both use only 3B + Curriculum. Yet according to Takeaway 1, Curriculum is precisely not the best reward at 7B. Thus the “Mixed 4:3:3” and “1K sweet spot” conclusions come from a configuration known to be suboptimal for 7B, with no evidence that they transfer to 7B. This is a concrete mismatch in conditions, not an abstract issue of unexplored interactions.

⑦ The environment-stability axis has only a figure, no table, one model size, and one noise type Reproducibility risks

The only axis of five without a table. It tests only 3B and only “globally random tool-call failures.” Real API failures include timeouts, partial responses, dirty data, and rate limits—random failure is “no information,” while dirty data is “wrong information”; the latter is more damaging to the policy. As engineering guidance, Takeaway 7 has very limited external validity.

⑧ Teacher performance is unreported, so whether RL exceeds the distillation ceiling is unknown Gaps in the argument

SFT trajectories come from DeepSeek-V3.2-Exp-Thinking, with a 24.0% rejection-sampling retention rate. Yet the paper never reports the teacher's score on TravelPlanner's official test set. Whether 7B-Sum's 62.8% means “student surpasses teacher” or “teacher was already better,”readers cannot tell. If the teacher already achieves 60%+, STAR's entire pipeline can be redescribed as “compressing a large model's capabilities into 7B,” downgrading RL from “creating capability” to “improving distillation efficiency.”

⑨ Gaps in My Own Analysis

• Figures 5/6/7/8 are raster images, so exact coordinates could not be extracted. Some environment-stability and data-scale values are missing and explicitly marked where relevant.
• I could not verify that STAR and Planner-R1 used exactly the same test split, so point ③ should be treated as a rough comparison of separately published numbers, not a strictly controlled comparison.
• I did not run the code; all judgments about implementation details come from the paper's descriptions.

5.1 TravelPlanner — 2024.02 (v4: 2024.10), ICML 2024 Spotlight

Problem addressed. When LLM agent evaluation generally relied on single-step tool calls or short QA, TravelPlanner constructed the first realistic planning benchmark requiring both long tool sequences and joint satisfaction of multiple constraints: 1,225 planning intents, nearly 4 million sandbox records, and a two-dimensional evaluation protocol. In one sentence, it turned the vague question “can LLMs really plan?” into the number 0.6%.

What gap remained? This is a purely diagnostic paper. It established 0.6% and qualitatively identified three failure modes, but did not intend to provide any training method. The remaining gap was exceptionally clear: how can that 0.6% be raised? More subtly, its two-dimensional, four-level evaluation protocol was itself an unused resource: the micro/macro decomposition was merely an evaluation tool in the original paper, and nobody recognized that it could directly serve as a density knob for the reward function.

How the core paper responds. STAR did two things: raised 0.6% to 62.8%; and directly reused TravelPlanner's evaluation protocol as reward primitives. All five reward components are native TravelPlanner evaluation components: the four reward-density levels were not invented; they were “read out” of the benchmark's evaluation protocol.

Transfer of key designs. Directly translating an “evaluation protocol” into a “reward function”—a redefinition of function rather than a technical improvement: the same formulas were a scorecard in 2024 and a training signal in 2026. This also explains why STAR's reward shaping is task-specific.

5.2 Search-R1 — 2025.03 (v5: 2025.08)

Problem addressed. It was the first systematic extension of R1-style pure RL training to interleaved reasoning and tool use. Two key technical choices: retrieved token masking(retrieved tokens are excluded from policy gradients to avoid learning to repeat them), and pure outcome-based rewards. Across seven QA benchmarks, Qwen2.5-7B improved by 41% relatively and 3B by 20%.

What gap remained? Two gaps, both fatal in long-horizon settings: (1) A single tool and short horizon—typical trajectories use 1–3 calls. Outcome rewards work only if initial base success is substantially above 0 (Qwen2.5-7B base achieves 18.1% on QA); on TravelPlanner, the same base achieves only 0.1%, causing a pure outcome reward to degenerate into zero gradients. (2) Reward density was never examined—because outcome rewards sufficed in its setting.

How the core paper responds. STAR's entire reward-density axis is essentially a direct challenge to Search-R1's implicit premise: the viability of outcome rewards depends on initial success, which depends on task horizon. Table 1's Success column reads 33.8 / 46.6 / 60.9 across three sizes—it works, but only after an SFT warm start raises success from 0.0% to 6.9%–19.7%. This is a specific boundary condition on the R1 narrative that “pure RL needs no SFT.” STAR also uses Search-R1 as an OOD baseline, but as Part III shows, 7B-SFT alone already wins, and the victory comes from distillation rather than RL.

Transfer of key designs.“Outcome reward as the default starting point” is inherited, but demoted to one of four options—an increase in abstraction from “method” to “design dimension.”

5.3 DAPO — 2025.03 (v2: 2025.05), ByteDance Seed

Problem addressed. Against a backdrop of hidden o1 / R1 technical details, DAPO fully released a large-scale RL system. Qwen2.5-32B base scored 50 on AIME 2024, exceeding the previous state of the art, DeepSeek-R1-Zero-Qwen-32B's 47, with only 50% of the training steps. Four core techniques: Clip-Higher(decoupled clipping bounds to prevent entropy collapse), Dynamic Sampling(filtering zero-gradient samples), Token-Level Policy Gradient Loss, Overlong Reward Shaping.

What gap remained? All four techniques were designed for single-turn, long-CoT mathematical reasoning. Every premise must be rechecked for multi-turn tool interactions: Clip-Higher prevents entropy collapse, but entropy in multi-turn agentic settings also comes from external randomness in tool responses, which may consume the added exploration space. DAPO does not explain how to distribute the soft Overlong Reward Shaping penalty across a 9.2-step trajectory. Most importantly, DAPO was validated entirely at 32B and never answered whether its techniques remain effective at 1.5B or are already unnecessary at 7B.

How the core paper responds. Two levels. Surface level: DAPO enters Table 4 as an algorithm option, yielding a size-dependent result of +6.8 pp at 1.5B and −4.4 pp at 7B. Deeper level: STAR's GRPO baseline already incorporates two DAPO techniques(KL-free and clip-high are defaults)—DAPO therefore serves as both Control group and infrastructure in STAR.

Transfer of key designs.Clip-Higher → STAR's default configuration (inherited unconditionally); Overlong Reward Shaping → “exclude from loss / retain for normalization” (changed form); Dynamic Sampling → implemented by STAR through “balanced data difficulty” at the data level rather than the algorithm level. The third is most interesting: the same problem—waste from zero-gradient samples—is solved algorithmically by DAPO and through data composition by STAR.

5.4 ARPO — 2025.07, ICLR 2026 Poster

Problem addressed. The closest predecessor to STAR's problem domain. Its key observation is a concrete empirical phenomenon: token entropy rises substantially immediately after a tool-call response—tool responses inject uncertainty. This motivates entropy-based adaptive rollout(dynamically increasing step-level sampling at high-entropy steps), combined with advantage attribution estimation. Result: better performance on 13 benchmarks, while using only half the tool-call budget of existing methods.

What gap remained? The entire design rests on the assumption that insufficient exploration is the primary bottleneck. This holds at the scales tested, mainly 7B and below, but ARPO does not vary model size to test the boundary of that assumption, implicitly treating smarter exploration as monotonically beneficial.

How the core paper responds. Table 4 provides a direct counterexample: 1.5B +7.4pp(+18.9% cost) → 3B −0.7 pp (+55.1% cost) → 7B −4.5pp(+48.6% cost). ARPO's gains already turn negative between 1.5B and 3B. STAR consequently proposes Takeaway 6: “the need for complex exploration mechanisms is inversely related to model capability.” This is an adversarial response rather than an inherited extension—STAR does not improve on ARPO, but sets an upper boundary on ARPO's applicability, and that boundary lies inside ARPO's own experimental scale range.

Transfer of key designs. Form of technical transfer: from “explicit step-level exploration scheduling” → “implicit capacity-driven exploration”.

5.5 Planner-R1 — 2025.09 (v2: 2025.10), ICLR 2026 Under review

Problem addressed. The work that most directly competes with STAR: the same benchmark and the same claim. Results: 56.9% final-pass rate with only 180 training queries; an 8B model matches 32B performance, with 3.5× compute efficiency and 1.5× memory efficiency; a 2.7× improvement over GPT-5's 21.2% baseline; and “broadly maintains or exceeds baseline” on OOD tasks including Multi-IF, NaturalPlan, and τ-Bench.

What gap remained?(1) Limited scale validation—only 8B and 32B, insufficient to characterize the trend's shape; (2) code not publicly available(I found no public repository), raising reproducibility concerns; (3) it claims reward shaping is the main lever but does not include RL algorithm choice as a control axis, so it cannot rule out gains arising from the algorithm rather than the reward.

How the core paper responds. STAR expands Planner-R1's two-point observation into a curve with three samples and adds algorithm, data, and environment axes beyond reward, making its validation a strict superset. Precisely for that reason, the lack of a head-to-head with Planner-R1 is a serious omission: published numbers give Planner-R1 56.9% from 180 queries and STAR 62.8% from ≈2,198 examples. If Planner-R1's sample efficiency holds, STAR's “comprehensive recipe” is outperformed by contemporary work on data efficiency—the very dimension STAR's Takeaway 4 presents as a core value.

Transfer of key designs. STAR fully reproduces and refines the finding that “dense process-level rewards work better for small models.” Transfer takes the form: from “a qualitative conclusion based on two points” → “locating a transition on a three-point curve”.

The internal logic of the evolution

The driving force can be summarized in one sentence—from “getting agents to act” to “getting agents to learn to act” to “knowing how to teach them”.

• Stage 1 (2023–2024.02): Demonstrating and falsifying capabilities. ReAct showed that prompts can make LLMs use tools; TravelPlanner's 0.6% showed that this route does not reach long horizons. Turning-point signal: 0.6% is too low to explain away as merely needing more prompt engineering.
• Stage 2 (2025.03–2025.07): Building the RL machinery. Search-R1 addresses interleaving RL and tools, DAPO addresses scaling without collapse, and ARPO addresses credit assignment in multi-turn settings. These are parallel infrastructure efforts that do not cite one another. Shared blind spot: everyone optimizes methods; nobody treats model size as an experimental variable.
• Stage 3 (2025.09–2026.03): Recipes and boundaries. Planner-R1 first asks what reward density fits what model size; STAR expands the question to five axes. Key turn: the object of study shifts from algorithms to interactions between algorithms and capacity.

Three Shifts in Community Attention

1. Accuracy → Data efficiency: Search-R1 reports relative gains, Planner-R1 reports “180 queries,” and STAR reports the “1K sweet spot.”Sample budgets move from background information to first-class metrics.
2. Performance → Performance/cost ratio: STAR's Table 4 puts GPU hours alongside Success; ARPO's “half the tool budget” is an early signal of the same trend.
3. In-domain → Generalization tax: this transition is not yet complete. STAR is among the first to systematically report OOD degradation, but as Part III shows, its own data support a stronger conclusion than it acknowledges. This is the next turning point in the lineage, and no paper has yet taken that position.

8.1 Theory

① Does RL's net OOD contribution monotonically approach zero with capacity? Where is the crossing?

An immediately executable experiment: at 1.5B / 3B / 7B / 14B / 32B, hold SFT and RL data fixed and measure only “OOD mean of the best RL configuration − SFT OOD mean”. STAR already gives +14.1 / +4.3 / +1.0 at three points; extrapolation predicts a negative value at 14B. If the crossing is indeed around 10B, industry needs to price in the OOD cost of agentic RL on 30B+ models. High value, low barrier—reuse STAR's open-source pipeline and add only two model sizes.

② Is weighting reward components more important than reward density?

STAR's r_sum five component coefficients are all 1. Table 3's Hard-1K setting shows CS Macro collapsing by 34.6 pp under data-distribution shift while HC components barely move—suggesting that effective component gradient weights are determined by the data distribution rather than reward coefficients. Concrete experiment: hold Mixed-1K fixed and sweep r = α·s_cs^micro + β·s_cs^macro + γ·s_hard^micro + δ·s_hard^macro + ε·s^success 's coefficient simplex, testing whether the Success response surface contains a point better than (1,1,1,1,1), and whether the optimum moves with size. This expands the one-dimensional reward-density knob to five dimensions and tests whether Takeaway 1 is a spurious transition caused by a low-dimensional projection.

③ The mechanism of “capacity substitutes for exploration” is missing

STAR establishes the phenomenon but not the mechanism. Operational test: measure within-group rollout trajectory diversity at 1.5B / 3B / 7B, such as variance in tool-sequence edit distances. If the hypothesis holds, GRPO's within-group diversity at 7B should already approach or exceed ARPO's diversity at 1.5B. This direct measurement needs no new training.

8.2 Engineering and Applications

④ A Controlled Planner-R1 vs. Agent-STAR Head-to-Head

The clearest experimental gap in the field. Suggested setup: identical base (Qwen2.5-7B), GPU budget, and test split; total supervised sample budget (180 / 500 / 1K / 2198) on the horizontal axis, Success and mean OOD on the vertical axis. The curves' crossing directly answers how far reward shaping's sample-efficiency advantage persists. Current separate numbers—56.9% @ 180 vs. 62.8% @ 2198—cannot distinguish method differences from budget differences.

⑤ A Benchmark of Environment-Noise Types

STAR tests only random global failure. Real tool failures have at least four types with different levels of harm:

Concrete opportunity: build a configurable noise-injection layer and sweep 0/5/10/20% for each of the four types, producing a 4×4 robustness matrix. Such environment-side benchmarks are currently absent, and the barrier is low: STAR's sandbox is open source.

⑥ Do Recipes Change in Online / Streaming Settings?

All existing work uses fixed offline datasets. Real agent deployments are streaming: tasks arrive continuously, and difficulty distributions drift over time. Concrete question: when difficulty distributions drift, does static “Mixed 4:3:3” remain optimal, or is adaptive difficulty sampling needed, such as inverse weighting by the current policy's success rate? This can be instantiated directly in STAR: divide 1K prompts into 10 batches presented in different difficulty orders and compare static mixing with adaptive sampling.

⑦ Quantifying the Teacher Ceiling

Following critique ⑧. Concrete experiment: report DeepSeek-V3.2-Exp-Thinking's Success on the official TravelPlanner test set and compute “distillation efficiency ratio” = 62.8% / teacher score. If the ratio is > 1, RL has created capabilities the teacher lacked; if < 1, the entire pipeline should be reframed as compression rather than training. Extremely low cost, but a major impact on interpreting the conclusions.

8.3 Emerging Directions

⑧ Reward Decomposition for Multimodal Long-Horizon Tool Orchestration

STAR's four-level density ladder assumes evaluation decomposes into binary checks. In multimodal agents, such as GUI interaction and embodied navigation, success is often continuous or requires a VLM judge. How should micro/macro decomposition be defined? Concrete question: When a noisy VLM judge produces the macro indicator, judge noise smooths its discontinuity. Does r_sum 's “finish the last step” incentive then fail? This must be answered before transferring STAR's methodology to embodied / GUI domains.

⑨ Do Scale-Dependent Recipes Also Apply to SFT?

STAR ablates model size only during RL; the SFT stage's 1,198 trajectories are exactly identical. Yet Table 1 shows very different post-SFT starting points (6.9 / 12.2 / 19.7). Concrete opportunity: should 1.5B use more but simpler SFT trajectories, and 7B fewer but harder ones? In other words, move the data-difficulty-mixture axis from RL upstream to SFT.

⑩ Forgetting and Safety Boundaries in Long-Horizon Agents

KL-free training lets the policy drift arbitrarily from the SFT distribution. What does the model lose while gaining 43.1 pp in-domain? STAR tests only QA-style OOD. Concrete opportunity: evaluate STAR checkpoints on safety / instruction following / refusal consistency, such as Multi-IF and harmful-instruction refusal rate, to quantify agentic RL's “alignment drift.” All six checkpoints are public. This experiment can be completed within a day, and no agentic RL paper currently reports this kind of data.

The following papers were in this issue's candidate pool but are not examined in depth. I have not read them closely; assessments rely on abstracts and search results. Consult the originals.

Agentic Reinforced Policy Optimization (ARPO) — 2025.07, ICLR 2026 Poster

Already discussed as related work, but still worth reading independently: the observation that token entropy rises after tool responses is a reusable diagnostic signal. It does not conflict with STAR's critique—ARPO's +7.4 pp at small sizes is real.
arXiv | Code | ICLR page

A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning — 2025.10, NeurIPS 2025

Like STAR, this is an empirical recipe study, but it emphasizes general multi-turn interaction rather than a single benchmark. Reading it alongside STAR can test whether scale dependence holds robustly across tasks—precisely the OOD limitation STAR acknowledges.
arXiv | NeurIPS page

Rethinking Agentic Reinforcement Learning In Large Language Models — 2026.04, arXiv

The title suggests a systematic challenge to agentic RL's basic assumptions, potentially echoing this report's derived finding that net OOD RL contributions approach zero. Prioritize checking whether it reports OOD controls.
arXiv

From Reasoning to Agentic: Credit Assignment in RL for LLMs — 2026.04, arXiv

Credit assignment is central to 9.2-step trajectories, whereas STAR's reward axis uses density to work around assignment. This paper addresses assignment directly and complements STAR.
arXiv | HF Papers

RL for LLM-based Multi-Agent Systems through Orchestration Traces — 2026.05, arXiv

Extends STAR's single-agent long-horizon problem to multi-agent orchestration. If scale dependence holds for a single agent, the appropriate size of each agent becomes a new design dimension in multi-agent systems.
arXiv

Benchmarking and Improving GUI Agents in High-Dynamic Environments — 2026.04, arXiv

Directly addresses the environment-noise taxonomy gap proposed in 8.2 ⑤, but for GUIs rather than tool APIs. It is an existing instance of that research opportunity; first check how far it goes.
arXiv

Finetuning VLA Models Requires Fewer Layers Than You Think — 2026.06, arXiv

Embodied intelligence. Shares STAR's argumentative structure: something assumed necessary may not be necessary. Read together, the papers reveal a common tendency toward subtraction in this wave of 2026 empirical studies.
arXiv

Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs — 2025.09

Already discussed in Part V; repeated here because it is the most direct comparison for testing STAR's conclusions, and if its 180-query sample-efficiency claim holds, the implications extend far beyond TravelPlanner.
arXiv | OpenReview