EARL-BO: in-depth revised report

Positioning in one sentence: EARL-BO no longer explicitly computes a rollout acquisition that grows exponentially with lookahead depth. Instead, it treats the current Gaussian process as a temporary world model, compresses the growing observation set with Attention–DeepSets, and uses PPO to learn a multistep query policy inside that model.

Date: 2026-08-13
Today's field: Reinforcement learning / agents / embodied intelligence (Thursday rotation)
Core paper: Mujin Cheon, Jay H. Lee, Dong-Yeun Koh, Calvin Tsay. EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimization. ICML 2025, PMLR 267:10182–10198.
Sampling record: 24-paper candidate pool; ran shuf -i 1-24 -n 1 and selected paper 10. The full text, official conference page, and v2 were accessible; no redraw was needed.
Evidence policy: Numbers in the main text come from v2/the ICML version. Main performance results appear only as curves, without pointwise numerical tables. This report does not estimate endpoints from pixels or fabricate a precise five-benchmark × six-method matrix; it instead provides a matrix of what the paper actually reports and its only exact runtime table.

Central assessment of this revision: EARL-BO's contribution is not that RL is stronger than BO. It shifts the computational bottleneck of multistep BO from expanding an exponentially growing scenario tree online to repeatedly training a finite-horizon policy in the current GP posterior. This makes 30D nonmyopic planning tractable in the paper's setting, but still requires 14–27 minutes of CPU planning before each real query, and the policy can amplify GP model bias. Its useful regime is therefore one where a real experiment is much more expensive than planning and the posterior is already sufficiently trustworthy—not general HPO.

0.0 Decision summary: start with the evidence table

This table also explains how to read the revision: exact numbers are reported only when provided by the paper; curves support qualified ordinal comparisons; authors' causal explanations are distinguished from observed patterns.

0. Link verification table

All links below were accessed when this report was generated. The currently readable arXiv version is 2411.00171v2, 2026-04-23; v1 dates to 2024-10-31.

0.1 Candidate pool (24 papers)

The candidate pool prioritizes ICML 2025, ICLR 2026, and high-quality preprints from 2025 onward, all within 24 months. Yesterday covered EPG (generative models), avoiding overlap with this issue.

1. Sleeping Reinforcement Learning (ICML 2025)
2. Flow Q-Learning (ICML 2025)
3. Enhancing Cooperative MARL with State Modelling and Adversarial Exploration (ICML 2025)
4. Meta-RL with Adaptation from Human Feedback (ICML 2025)
5. Test-time Adapted RL with Action Entropy Regularization (ICML 2025)
6. Policy Regularization on Globally Accessible States in Cross-Dynamics RL (ICML 2025)
7. Multi-objective Linear RL with Lexicographic Rewards (ICML 2025)
8. Logarithmic Regret for Online KL-Regularized RL (ICML 2025)
9. Unveiling Markov Heads in Pretrained Language Models for Offline RL (ICML 2025)
10. EARL-BO (ICML 2025, selected)
11. Adaptive Sample Sharing for Multi-Agent Linear Bandits (ICML 2025)
12. BiAssemble (ICML 2025)
13. Embodied Lifelong Learning Agents with Non-Parametric Memory (ICLR 2026)
14. ERA: Transforming VLMs into Embodied Agents via RL (ICLR 2026)
15. MEAL: Continual MARL Benchmark (2025)
16. Balancing Specialization and Centralization (2025)
17. Active Reward Modeling (ICML 2025)
18. Reducing Tool Hallucination via Reliability Alignment (ICML 2025)
19. SAC-GLAM (ICML 2025)
20. RL Finetunes Small Subnetworks in LLMs (ICML 2025)
21. Efficient Online RL Fine-Tuning Need Not Retain Offline Data (ICML 2025)
22. The Power of Context in Image-based RL (ICML 2025)
23. Decoupling Exploration and Exploitation in RL (ICML 2025)
24. Robust Reward Modeling via Causal Rubrics (ICML 2025)

1. In-depth analysis of the core paper

1.1 Motivation: the question is whether looking ahead remains computationally affordable

Classical Bayesian optimization (BO) fits a Gaussian process (GP) to an expensive black-box function, then chooses the next query through an acquisition function such as Expected Improvement (EI). Its strength is sample efficiency; its weakness is optimizing only the next-step reward. The paper sharpens this point: when today's choice changes tomorrow's GP posterior, next-step optimality does not imply optimality under a finite budget. BO is inherently a finite-horizon stochastic dynamic program.

Multistep rollout accounts for future queries in expected cumulative reward, but incurs two nested costs: every candidate action samples possible observations, and each observation changes the posterior and triggers another acquisition optimization. Cited predecessors demonstrated only around 1–4 dimensions; Rollout_VR can reach 6 steps but cannot run in this paper's high-dimensional experiments. Appendix Table 2 supplies the clearest numerical evidence: on 8D Ackley, three-step Rollout_EI exceeds 3,600 seconds per BO iteration even with only 1,000 Monte Carlo samples; three-step EARL-BO takes 840±147 seconds and five-step takes 1,075±134 seconds. Thus three-step planning saves at least 76.7% of the time (the rollout timeout is only a lower bound, so the actual saving is greater), but remains about 3,111 times slower than TuRBO's 0.27 seconds.

The paper therefore addresses a specific three-way tension:

• nonmyopic decisions require multistep planning;
• high dimensionality rapidly expands states, actions, and rollout branches;
• black-box evaluations are expensive, so RL cannot rely on extensive real-world trial and error.

EARL-BO retrains a policy within every real BO iteration, letting it interact only with the virtual world defined by the current GP posterior. The real function is queried once; the many RL trajectories come from the surrogate.

1.2 The paper's storyline: four turning points

Turning point 1: recast acquisition-function selection as an MDP. The state is the current dataset (D_t), the action the next query (x_{t+1}), the transition adds an observation, and the reward is improvement. Designing an acquisition function becomes learning a policy.

Turning point 2: raw states cannot be fed directly into RL. (D_t) grows with each iteration, while the order of set elements should not affect decisions. Attention–DeepSets encodes arbitrarily sized and ordered observation sets into fixed 16-dimensional vectors. It trades recoverable pointwise information for a fixed state dimension and permutation invariance.

Turning point 3: the real function cannot serve as the RL environment. The current GP posterior serves as a world model, sampling virtual observations from (N(^k(x),K k(x,x))) to produce 4,000 training episodes. Policy optimality is tied to GP calibration rather than the real function—the source of the planning delusion discussed later.

Turning point 4: purely on-policy learning starts too slowly. TuRBO supplies actions for the first 400 episodes to pretrain the actor/critic. Two layers are then frozen, and PPO and the encoder are updated end to end. This first imitates a strong one-step baseline and then seeks to surpass its myopia, rather than learning acquisition from scratch.

1.3 Formal definitions and mechanism analysis

Given observations (D_k={(x_i,y_i)}_{i=1}^k), the GP posterior is:

[ ^{k(x)=(x)+k(x)}(K+^2I){-1}(y-(x)), ]

[ K^{k(x,x)=K(x,x)-k(x)}(K+^2I){-1}k(x). ]

The finite-horizon policy objective (Equations 3–4) is:

[ V_h^(s_0)=E, ^*=_{}V_h^(s_0). ]

A common immediate reward is positive improvement:

[ R(D_t,x_{t+1},D_{t+1})=(0,y_{t+1}-y_t^*). ]

Including remaining steps in the value function gives the dynamic-programming recursion:

[ Q_h(D,x)= E_{yp(x,D)} , ]

[ V_h(D)=_{xX}Q_h(D,x),V_0=0. ]

Traditional rollout/one-shot trees explicitly approximate this expectation and nested maximization before each real query. EARL-BO instead trains (_(x(D))) to amortize this argmax. It does not eliminate the Bellman problem; it replaces expanding futures for every candidate with generating 4,000 finite-horizon episodes in the GP and optimizing a policy. This distinction matters: the former incurs scenario-tree/MC and inner-optimization errors, whereas the latter additionally incurs representation error, policy-optimization error, and model exploitation.

A notational detail must also be resolved during reproduction: the method describes maximization using ((0,y_{t+1}-y_t^)), but the synthetic-experiment text says the objective is to minimize these functions, and defines simple regret as (y_{opt}-y_k^). An implementation must specify whether to negate the objective first or change improvement to ((0,y_t^*-y_{t+1})). Neither the main text nor appendix provides code for this conversion.

Attention–DeepSets (Equations 6–8) encodes individual points, then aggregates them invariantly with normalized attention:

[ i=(f{att}((x_i,y_i))), _{att}(D_t)=(_i_i(x_i,y_i)). ]

PPO uses a clipped surrogate (Equation 5), while the critic uses MSE (Equation 10). The pretraining loss (Equation 11) combines PPO, value, and entropy terms:

[ L_{pre}=E_{D_{baseline}}[L^{PPO}+c_1L{VF}-c_2H(_)]. ]

The algorithm can be summarized as:

Real data D_k ──fit──> GP posterior (temporary world model)
     │                         │
     └─> Attention–DeepSets ─> PPO actor/critic
                                 │
           First 400 episodes: TuRBO actions (off-policy warm start)
           Later episodes: PPO actions + virtual GP observations (on-policy)
                                 │
                         Output one real query x_{k+1}

A more precise training schedule

Outer loop: real BO iteration k
  1. Refit an RBF+WhiteKernel GP on real observations D_k
  2. Initialize encoder, actor, critic, and buffer
  3. Run up to 4,000 virtual episodes, each reset to the same D_k
       First 400 episodes: TuRBO supplies actions
       Remaining 3,600 episodes: PPO actor supplies actions
       Repeat H times within each episode:
           a_t ← policy / TuRBO
           y~ ← GP posterior conditioned on current virtual data
           D_virtual ← D_virtual ∪ {(a_t, y~)}
           r_t ← improvement
       Update networks every 50 episodes; freeze the first 2 actor/critic layers after off-policy training
  4. If mean reward=0 for 15 consecutive updates, or final mean reward<1e-5: return TuRBO's suggestion
     Otherwise: the final actor outputs x_{k+1} from the encoding of real D_k
  5. Only now query the real black box once, obtaining y_{k+1}, then begin the next iteration

Initializing every iteration is the literal procedure in Algorithm 1. The paper does not state whether weights are reused across BO iterations. Complete retraining makes 840 seconds a recurring cost per real sample; weight reuse would mean the pseudocode omits crucial state transfer. The unavailable core code prevents independent resolution.

1.4 Experimental design

Synthetic tasks. Ackley, Levy, Rosenbrock, and Sum Squares; dimensions 2, 5, 8, and 30; common domain ([-15,15]^d); 30 random initial points; 10 repeats per setting; simple regret as the metric. Random, EI, TuRBO, and SAASBO participate in low-/mid-dimensional comparisons; Rollout_VR is feasible only in 2D. Random is omitted from the 30D plot for readability.

Real HPO. HPO-B search spaces 5889 (6D), 5968 (8D), and 7200 (19D). The 6D/8D tasks use 5 initial points; the main 19D experiment uses 50. There are 10 repeats, and PI is added as a baseline. The appendix reduces the 19D initialization to 5 points to probe planning delusion with a sparse surrogate.

Two reporting issues were insufficiently highlighted in the earlier version. First, Figure 4's legend lists UCB, EI, PI, EARL-BO, TuRBO, SAASBO, whereas the main text lists random search, EI, PI, TuRBO, and SAASBO. Code is needed to determine whether the first curve is UCB or Random. Second, Appendix A.2 lists EI, Random, TuRBO, and Rollout-VR as synthetic baselines but omits SAASBO, which appears in the main plots. Neither changes the plotted EARL-BO curve, but both matter for exact reproduction.

The total is 16 synthetic settings (4 functions × 4 dimensions) + 3 HPO-B search spaces + 1 sparse-data stress setting. Evaluation breadth should not be confused with the number of independent real tasks: the 16 synthetic settings share four analytic functions. Only three HPO-B search-space IDs are selected, without specific learner/task names, task IDs, or discrete candidate-set sizes. The official HPO-B repository allows multiple datasets/tasks under one search space, so search-space IDs alone do not uniquely identify the experiments.

1.5 Main results: what the paper actually reports

The paper provides no table of curve endpoints, AUCs, or average ranks, and no machine-readable CSVs/code. The following matrix therefore retains only ordinal conclusions clearly supported by Figures 2–4, without presenting visual estimates as exact numbers.

Claim–evidence audit matrix

The most persuasive exact evidence concerns runtime rather than performance endpoints:

Moving from H=3 to H=5 adds 235 seconds / 28.0%; against the rollout timeout lower bound, H=3 is at least 4.29× faster. This establishes a solution to intractable traditional multistep rollout, not production-ready high-throughput BO. The paper's slides also acknowledge roughly 850 seconds per step.

Treating the 10 repeats in Table 2 as independent samples permits approximate 95% t intervals for interpreting variability (t_{0.975,9}=2.262): EI 0.28±0.036s, TuRBO 0.27±0.143s, SAASBO 168.6±19.7s, EARL-BO H=3 840±105s, and H=5 1075±96s. The paper performs no runtime significance test; these intervals are recalculated here, not originally reported.

Cost break-even: when is 14 minutes of planning worthwhile?

Compared with TuRBO, EARL-BO H=3 adds approximately (840−0.27=839.73) seconds per iteration, or 13.996 minutes; H=5 adds 17.912 minutes. If EARL-BO reliably saves at least one real evaluation relative to TuRBO, wall-clock savings are possible only when that evaluation costs more than this threshold. The paper reports neither evaluations needed to reach the same regret nor numerical hitting times, so an actual break-even cannot be calculated; this is only a necessary condition.

For 30D, the paper reports only approximately 1600 seconds for EARL-BO and increases of 533%/215% over 8D for EI/TuRBO. Recalculating from Table 2 gives about 1.77s and 0.85s. Even with a gentler dimensional scaling ratio, EARL-BO's absolute overhead remains approximately 903 times EI's and 1881 times TuRBO's. “Scales better” must therefore be distinguished from “is faster.”

A more realistic total cost is:

[ T_{total}(B)=_{k=1}^{B}. ]

EARL-BO is practically attractive only when (T_{realeval}T_{policy}). HPO-B's table-lookup benchmark does not meet this premise, so Figure 4 establishes the quality of the sample sequence, not an end-to-end HPO wall-clock advantage.

1.6 Ablations, sensitivity, and regime changes

Regime changes. The key finding is nonmonotonicity, not that longer horizons always help: H=1 leads early but finishes worst; H=3/5 form a sweet spot; H=7 deteriorates as model errors accumulate. A sharper transition follows information density: reducing initialization from 50 to 5 points in 19D eliminates the multistep policy's advantage. Effective planning depth should therefore depend on posterior calibration/uncertainty rather than remain fixed.

Statistical significance. All main curves show means ±1 standard deviation over 10 repeats; appendix runtimes also include standard deviations. However, the paper reports no confidence intervals, paired tests, p-values, effect sizes, or cross-task average ranks. “Significantly” therefore primarily describes visual separation rather than statistical testing.

1.7 Actual contributions and reusable designs

The contribution goes beyond applying PPO to BO, which earlier RL-BO work already did. It connects three interfaces: set-state encoding, a virtual GP environment, and a strong BO-policy warm start. Together, these extend an RL planner previously limited to regular grids/low dimensions to 30D, learning specifically for the current posterior inside each BO iteration without offline cross-task meta-training data.

Three designs can be reused independently:

1. A set-valued belief state. Any agent whose state is a growing unordered observation set can use attention plus invariant pooling, avoiding padding to a fixed history length.
2. A strong heuristic warm start plus a learnable residual policy. TuRBO behavior avoids a random start; subsequent on-policy learning targets long-term reward. This suits planning agents in expensive environments.
3. World-model confidence controls planning depth. Although unimplemented, the planning-delusion evidence directly motivates an uncertainty-adaptive horizon: fall back to H=1/TuRBO when the posterior is unreliable, and increase to H=3/5 when trustworthy.

1.8 Limitations

Stated or directly acknowledged by the paper

1. High computational cost. Appendix C and the slides specify about 840–1,075 seconds per step and roughly 25 hours per complete experiment (AMD EPYC 7742, 16 CPUs, up to 100GB).
2. planning delusion. Section 4.1 notes that long horizons accumulate GP errors; Figure A2 further shows the loss of advantage in 19D with 5 initial points.
3. Longer horizons do not deliver monotonic gains. H=3 and H=5 are similar, while H=7 degrades. The paper acknowledges that a policy may be more sensitive to an incorrect world model than heuristic rollout.

Additional critique (10 independent observations)

1. Experimental-design issue: the performance matrix is not auditable. Figures 2–4 provide no endpoint values, AUCs, hitting times, or CSVs, preventing recalculation of mean gains, paired tests, or cross-task ranks. Clear superiority can currently mean only visual curve separation, not statistical significance.
2. Argument gap: competitor sets differ between low and high dimensions. Rollout_VR appears only in 2D, while high-dimensional comparisons use only myopic methods. Thus superiority to multistep methods and scalability lack a high-dimensional head-to-head test. Table 2's timeout establishes that one rollout implementation is slow, not that all approximate nonmyopic methods are excluded.
3. Missing key competitor: One-Shot Multi-Step Trees is not evaluated. Jiang et al. 2020 use joint scenario-tree optimization and GP fantasization for 2–4-step nonmyopic BO. Since EARL-BO claims to replace tree computation, it should compare directly under matched H and MC/CPU budgets, rather than mention this only in related work.
4. Asymmetric compute budgets. EARL-BO uses up to 4,000 episodes before each real query, taking 840s at H=3 versus TuRBO's 0.27s. Controlling only evaluation count systematically favors the planner. When a black-box evaluation costs less than about 14 minutes, even saving one evaluation may not reduce total time.
5. Underperforming subtask: a structural disadvantage on Sum Squares. SAASBO beats EARL-BO on 2/5/8D Sum Squares. The claim that nonmyopia becomes more valuable with dimensionality therefore depends on function structure. In additive, axis-aligned landscapes, an appropriate structural prior matters more than deeper planning.
6. Confounded attribution: TuRBO fallback on failure. After 15 consecutive updates without reward, or when final mean reward is<1e-5, EARL-BO directly returns a TuRBO query. The paper does not report fallback counts by task/seed, so the EARL curve is an unknown mixture of learned and teacher policies. Its gains and stability cannot all be attributed to PPO.
7. Reproducibility risk: core code and algorithmic state transfer are unavailable. The actor distribution, action bounds/squashing, layer counts, advantage/return computation, reward normalization, GP refitting, seeds, and whether each iteration restarts from scratch can all materially affect results. Algorithm 1 and Table 1 alone cannot reproduce a 25-hour experiment.
8. Internal documentation inconsistencies. The main experiments explicitly use H=3 and additionally test H=5 in 8D, while Appendix Table 1 lists only Horizon=5. Figure 4's legend says UCB but the text says Random, and the appendix baseline list omits SAASBO. The paper alone therefore cannot establish one consistent reproduction configuration.
9. Metric-validity issue: simple regret alone. Real applications also care about cumulative regret, wall-clock regret, failure rates, parallel-batch efficiency, and constraint violations. The paper plots only simple regret; its 3,111× TuRBO latency is absent from aggregate utility, and table-lookup HPO-B tasks include no real training cost.
10. Theoretical and model risk: “near-optimal” lacks a testable definition. There is no oracle DP, policy-value gap, approximation ratio, or regret bound. Meanwhile, the world model uses only an RBF+White GP, and the sparse 19D stress test already shows model error can reverse the advantage. “Near-optimal” is therefore motivational wording rather than an established property.

2. Related work: how the problem reaches EARL-BO

2.1 Efficient Rollout Strategies for Bayesian Optimization — UAI 2020

Problem solved. Lee et al. reduce the variance of expensive high-dimensional rollout integrals through quasi-Monte Carlo, common random numbers, and control variates, using policy search to avoid direct rollout-acquisition optimization. This advances nonmyopic BO from concept toward a runnable implementation.

Remaining gap. Even with variance reduction, computation rises sharply with dimension and horizon. In EARL-BO Table 2, its three-step implementation exceeds 3,600 seconds with 1,000 MC samples and cannot enter the main 5/8/30D comparisons.

The focal paper's response. Rather than expanding a future tree for every candidate action, EARL-BO trains an amortized policy, approximating the dynamic-programming solution through trajectory updates in the GP environment.

Inherited design. The framework of multistep cumulative improvement and GP-posterior future simulation remains; actor–critic replaces numerical integration and the heuristic base policy.

2.2 Efficient Nonmyopic BO via One-Shot Multi-Step Trees — NeurIPS 2020

Problem solved. Jiang et al. reparameterize nested future action/observation optimization as joint optimization over a full scenario tree. GP fantasization and automatic differentiation enable general multistep EI without repeatedly solving inner argmax problems at each level. This alternative still optimizes acquisition but flattens the nested structure into one problem.

Remaining gap. Branch count still depends on the product of fantasy-sample counts across levels, while optimization variables and GP updates grow with horizon. NeurIPS review records indicate experiments reached about four steps, with results sensitive to branching factor, warm starts, and optimizer choice. The method eases tree computation but neither compresses arbitrary histories to fixed states nor learns a directly reusable continuous policy.

The focal paper's response. EARL-BO replaces joint candidate-tree optimization with an actor: the tree is no longer stored explicitly, and future samples enter the buffer only as PPO training trajectories. Empirical dimension/horizon scaling becomes gentler, but numerical-integration error is replaced by policy, representation, and model errors.

Inherited design. Both use GP-posterior fantasies to represent future uncertainty. One-shot optimization treats all future actions as optimization variables, while EARL-BO parameterizes the action rule as (_). This is the head-to-head comparison the paper most needs but does not perform.

2.3 TuRBO — NeurIPS 2019

Problem solved. TuRBO uses local trust regions and bandit-style global allocation to avoid overexploration by a single global GP in high dimensions. The predecessor reports scaling to 200D and outperforming conventional global BO on tasks including robotic control.

Remaining gap. It remains primarily a one-step acquisition method. Locality improves high-dimensional efficiency without explicitly optimizing the long-term effect of current decisions on future posteriors.

The focal paper's response. EARL-BO treats TuRBO as a teacher for the first 400 episodes, rather than a competitor to discard, then uses PPO to learn multistep cumulative reward.

Inherited design. The trust-region query policy shifts from the final algorithm to a behavioral prior and fallback—a transfer resembling distillation from a strong baseline.

2.4 SAASBO — UAI 2021

Problem solved. SAASBO places a strong sparsity prior on inverse length scales and uses HMC to identify a few relevant axis-aligned dimensions, maintaining sample efficiency in searches with hundreds of dimensions without problem-specific hyperparameters.

Remaining gap. The sparse axis-aligned assumption does not suit every complex coupled landscape, and acquisition remains myopic.

The focal paper's response. Instead of directly selecting a relevant subspace, EARL-BO learns observation-level weights with an attention encoder and plans actions with PPO. It wins on four 30D functions but loses to SAASBO on additive Sum Squares.

Inherited design. Both reduce high-dimensional difficulty by focusing on relevant information, but SAASBO applies Bayesian sparsity to coordinates while EARL-BO compresses a set of observations through attention.

2.5 Non-myopic BO using model-free RL — 2024

Problem solved. Cheon et al. directly learn a multistep BO policy with model-free RL, demonstrating feasibility in 2D and electrochemical optimization while bypassing rollout base-policy approximation bias.

Remaining gap. They discretize/rasterize the GP on a regular grid for the state. Grid size grows exponentially with dimension, limiting extension beyond low-dimensional settings.

The focal paper's response. EARL-BO represents (D_t) with fixed-dimensional Attention–DeepSets and outputs query points through a continuous-action actor, extending nonmyopic RL to high dimensions.

Inherited design. The RL policy and improvement reward remain; explicit GP grids become learned set representations, and model-free trial and error becomes GP-based simulation.

3. Technical evolution

2016 approximate dynamic-programming BO [finite budget; Gauss–Hermite; low dimensions]
   ├── 2019/2020 rollout / one-shot trees [multistep; variance reduction or differentiable trees]
   │          └── Gap: trees and integrals grow with dimension and horizon
   ├── 2019 TuRBO [local trust regions; hundreds of dimensions; still myopic]
   ├── 2021 SAASBO [sparse axis-aligned subspaces; still myopic]
   └── 2024 RL-BO [policy learning replaces the base policy; GP grids limit dimensionality]
                  ↓
2025 EARL-BO [set-state encoding + TuRBO warm start + GP world model + PPO multistep policy]
                  ↓
Next: posterior-calibrated adaptive horizons + wall-clock-aware rewards

The underlying shift is from maximizing an acquisition function at every step to learning a decision rule reusable across future states. The first stage defines nonmyopia; the second addresses high-dimensional locality and sparsity; the third uses RL to remove explicit rollout trees. The turning point is the GP becoming an RL world model rather than merely a surrogate. As optimization power grows, model bias also escalates from prediction error to policy error.

3.1 Reproducibility blueprint: actually rebuilding this paper

The core code is unavailable, so the following is not the authors' implementation. It is a minimal reproduction protocol compiled from Algorithm 1, Appendix A, and Table 1, with every unresolved field explicitly identified.

A reproduction should provide four machine-readable tables, not just redraw the original figures:

1. regret_by_seed.csv: task, dimension, method, seed, iteration, simple_regret;
2. runtime_breakdown.csv: GP fit, TuRBO warm-start, PPO rollout, PPO update, final action;
3. fallback_log.csv: trigger reason, teacher query, actor query, and selected source at every iteration;
4. calibration.csv: GP posterior coverage, NLL, horizon, and final regret.

The third table is particularly important: without fallback logs, it is impossible to determine whether EARL-BO is an independent planner or a hybrid of a TuRBO safety net with occasional PPO use.

4. Open challenges and actionable research opportunities

4.1 Theory

1. Bounds from calibration error to policy regret. Fix the real evaluation budget and compare H=1/3/5/7 under known GP miscalibration (). Derive an upper bound on model-bias accumulation with horizon; empirically report both posterior coverage and final regret.
2. Adaptive horizons. Choose H using posterior entropy, calibration error, or ensemble disagreement. Compare against fixed H=1/3/5/7 with equal CPU and evaluation budgets, testing whether the policy automatically becomes myopic in 19D with 5 initial points.
3. Teacher dependence. With identical networks and seeds, replace the first 400 episodes with EI, PI, TuRBO, SAASBO, or Random to test whether the final policy is merely a local extension of its teacher.

4.2 Engineering and applications

1. wall-clock-aware objective. Change reward to improvement / (black-box cost + planner cost), sweeping evaluation cost from 1 second to 1 day to find EARL-BO's break-even point, explicitly accounting for TuRBO's 0.27 seconds and EARL-BO's 840 seconds.
2. Reuse instead of retraining at every step. Compare 4,000-episode retraining per BO step, incremental training from the previous checkpoint, and a cross-task meta-policy with limited adaptation. Control total gradient steps and report both latency and regret.
3. Stronger world models. Compare RBF GP, Matérn GP, deep kernels, heteroscedastic GP, and ensembles under equal data budgets. Test whether planning delusion worsens on nonstationary/discontinuous tasks.
4. Code and data protocols. Release per-iteration CSVs for every curve, 10 seeds, timeout rules, and hardware configurations. Report means, 95% CIs, paired tests, AUC-regret, and wall-clock regret.

4.3 Emerging directions

1. Agent tool selection. Replace black-box queries with expensive tool calls/experiments, retaining an unordered experience-set state. Compare Attention–DeepSets with trajectory Transformers to test whether set invariance discards causal order.
2. Batch and asynchronous BO. Compare q-EI, asynchronous TuRBO, and multi-agent EARL-BO on identical hardware, explicitly modeling parallel experiment return times. Single-action PPO does not yet address real laboratory parallel scheduling.
3. Safe planning. Add constraint-violation probabilities and compare penalty rewards, Lagrangian PPO, and safe acquisition. Report cumulative violations rather than only the final optimum.

5. Other recent work worth watching

Flow Q-Learning (2025.07, ICML)
Uses a flow-matching policy to represent multimodal action distributions in offline data, combining generative policies with value optimization. Unlike EARL-BO, it optimizes a fixed offline MDP rather than a GP world model that changes each iteration.
PMLR

Sleeping Reinforcement Learning (2025.07, ICML)
Incorporates actions becoming available or unavailable over time into RL, offering useful ideas for tool agents, resource-constrained scheduling, and dynamic action spaces.
PMLR

Enhancing Cooperative MARL with State Modelling and Adversarial Exploration (2025.07, ICML)
Uses state modeling and adversarial exploration to address partial observability and insufficient exploration in cooperative MARL, extending pure value decomposition toward learned state models.
PMLR

ERA: Transforming VLMs into Embodied Agents via Reinforcement Learning (2026, ICLR)
Trains a compact VLM into an embodied reasoning agent in two stages. It is worth examining how RL converts visual reasoning into closed-loop action.
OpenReview

MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning(June 2025, preprint)
Introduces continual learning to MARL, emphasizing long-term adaptation to changing environments/teammates rather than cooperation within a single training distribution.
arXiv

Balancing Specialization and Centralization(October 2025, preprint)
Compares multi-agent division of labor with centralized control for sequential industrial control, grounding MARL architecture choices in real control constraints through a recent benchmark.
arXiv

6. Final assessment

EARL-BO has clear research value: it connects the historically separate high-dimensional and nonmyopic BO directions, offering a structural template for agents planning expensive tool calls or experiments. Engineering maturity remains limited: it is at least 4.29× faster than traditional rollout but about 3,111× slower than TuRBO. Unreleased code and main-curve values also prevent independent calculation of whether performance gains justify the compute cost.

The central scientific finding is not that RL necessarily beats acquisition functions, but that planning depth must match world-model reliability: H=3/5 outperform H=1, while H=7 deteriorates; reducing 19D initial points from 50 to 5 eliminates the advantage. The next meaningful advance should adapt horizon, teacher, and compute budget to posterior calibration instead of treating 4,000 fixed episodes as the default answer.

Index status: read and updated ml-report-index.md.
A candid note: The focal paper provides neither code nor pointwise performance data, and this report does not estimate curve values from pixels. The inability to supply an exact percentage-performance matrix follows from the source's reporting granularity; speculative numbers are not substituted. HTML structure was validated, but the current environment had no executable Chromium in /usr/bin/chromium or /opt/pw-browsers , so browser-screenshot QA cannot be claimed. HTML validation and static overflow-rule checks were used instead.