ATLANTIS: Turning What a Small Model Learned into Large-Model SFT Weights—Is This Really Importance Sampling?

0. Verified links and evidence boundaries

These URLs were visited on 2026-09-07. All numbers come from PDFs downloaded and read page by page; web pages verify venues, version dates, and public assets. Core evidence follows the 11-page ACL Anthology final paper, proceedings pp.1042–1052. Exact-title searches found neither a core arXiv entry nor author code; these are marked not found, not nonexistent. OpenReview forums and attachments are reachable, but anonymous submissions do not replace the ACL final version. No dedicated entry was found through exact-title searches of Hugging Face Daily Papers or Papers with Code; Semantic Scholar has an official record.

Index cross-check: the core paper can be found in Semantic Scholar ; OpenReview and the official ACL page point to the same title. Candidates were first collected from ACL 2025 Main proceedings and ACL Anthology, then checked through arXiv, OpenReview, Hugging Face Daily Papers, Papers with Code, and Semantic Scholar for versions, code, and duplicates. Aggregator summaries were not numerical evidence.

1. Candidate pool, deduplication, and random selection

First read ml-report-index.mdto exclude closely related topics and papers covered in the preceding 14 days, including Alignment Potential preference-data selection, AVUT multimodal evaluation, and Emergent Misalignment from narrow fine-tuning, as well as nearby systems, vision, and generation papers. All candidates are ACL 2025 Main long papers within 24 months. The complete three-column list is saved in candidates_2026-09-07.tsv. Running shuf -i 1-24 -n 1returned 18, selecting ATLANTIS. The final paper contains main matrices, combinations with data selection, cross-architecture and scale tests, and proxy-tuning controls, providing sufficient material, so no redraw was needed。

1. M‑RewardBench: Evaluating Reward Models in Multilingual Settings
2. ELABORATION: A Comprehensive Benchmark on Human‑LLM Competitive Programming
3. Intuitive Fine‑Tuning: Towards Simplifying Alignment into a Single Process
4. StrucText‑Eval: Evaluating LLM Reasoning in Structure‑Rich Text
5. GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization
6. Tree‑of‑Evolution: Tree‑Structured Instruction Evolution for Code Generation
7. FACT‑AUDIT: Adaptive Multi‑Agent Fact‑Checking Evaluation
8. Capture the Key in Reasoning to Enhance CoT Distillation Generalization
9. UniICL: Unifying Compression, Selection, and Generation
10. RuleArena: Rule‑Guided Reasoning in Real‑World Scenarios
11. Semantic Exploration with Adaptive Gating
12. JuStRank: Benchmarking LLM Judges for System Ranking
13. Efficient and Accurate Prompt Optimization with Memory
14. BPP‑Search: Tree‑of‑Thought for Mathematical Modeling
15. EvoWiki: Evaluating LLMs on Evolving Knowledge
16. ProcessBench: Identifying Process Errors in Mathematical Reasoning
17. Model Extrapolation Expedites Alignment
18. ATLANTIS: Weak‑to‑Strong Learning via Importance Sampling
19. HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
20. Wait, that’s not an option: Robustness with Incorrect MCQ Options
21. MARS: Metaphysical Reasoning Benchmark
22. Ask‑Before‑Detection: Conformity Bias in Math Error Detection
23. Disentangling Memory and Reasoning Ability in LLMs
24. UnSeenTimeQA: Time‑Sensitive QA Beyond Memorization

2. Motivation: SFT involves more than identifying high-quality examples

Ordinary SFT minimizes negative log-likelihood on training distribution p_d , whereas the desired coverage concerns an intractable real-world language distribution p*. The problem is not merely removing poor examples. Even individually high-quality human or synthetic data can differ from deployment intopics, expression, difficulty, and preferences. Expanding data continuously is costly, while quality-based deletion may not correct directional bias (core PDF p.1 / proceedings p.1042, §1).

ATLANTIS converts this unobservable problem into a computable proxy. Given a small base model p_b^S and its official instruction-tuned version p_r, a large increase in an answer's probability is interpreted as movement toward a preferable direction. A same-family large base model p_b^L supplies a scale correction. Each training pair receives an influence weight without training a reward model or discarding most examples.

Two comparisons motivate this quantitatively. First, across Llama2-13B, Qwen2-7B, Gemma2-9B, and eight metrics, ATLANTIS exceeds vanilla SFT in 23/24 comparisons. The sole decline is Qwen2-7B Arena-Hard-Auto,9.9→9.5（−0.40 pp，−4.04%）(core PDF p.5 / p.1046, Table 2; recalculated here). Second, the appendix proxy-tuning control gives Qwen2-7B MT-Bench pairwise scores of SFT 25.63、proxy‑tuning 7.91、ATLANTIS 26.25, suggesting that using small-model shifts astraining weightsis far stronger here than adding logits at decoding (core PDF p.11 / p.1052, Table 7). One model and two benchmarks, however, do not establish universal superiority.

3. Argument, core assumptions, and trade-offs

The argument has four steps. Define the ideal training objective under p* . Use importance sampling to rewrite its inaccessible expectation under an available reference distribution p_r. Because p*/p_r remains intractable, borrow proxy-tuning's assumption thatprobability shifts transfer across scale: the small model's base-to-tuned change is proportional to the large model's base-to-ideal change. Finally, turn the ratio into per-example weights and update the large model normally (core PDF pp.2–4 / pp.1043–1045, Equations 1–8, Algorithm 1).

Three assumptions are required:

• Target: A unified optimal real-world distribution p*can represent knowledge, instruction following, and human preferences simultaneously. Actual users and tasks often conflict; the paper does not specify the measure space or target population.
• Proportional transfer: p*/p_b^L ∝ p_r/p_b^S. This is an empirically motivated proxy-tuning assumption, not a theorem proven here.
• Data source: Training responses can be treated as samples from p_r . In fact, training uses OpenHermes-2.5's mixed synthetic corpus, while p_r is the conditional distribution of an official chat/instruct model; they are not inherently identical.

ATLANTIS retains examples and combines with IFD or Superfiltering, but preprocessing requires large-base, small-base, and reference-model sequence probabilities for every example. Training adds only a scalar multiplication, whereasdata preprocessingadds three forward passes. No tokens/s, GPU-hours, cache sizes, or total wall-clock costs are reported, so acceptable overhead remains an author judgment (core PDF p.9 / p.1050, Limitations).

4. Formalization: from exact identities to proxy heuristics

The ideal objective is:

L*(θ) = − E_{x~q, y~p*(·|x)} log p_θ(y|x)                         (1)

Ordinary SFT substitutes the actual data distribution p_d :

L_SFT(θ) = − E_{x~q, y~p_d(·|x)} log p_θ(y|x).                  (2)

If p_r(y|x)>0 overwritten p*(y|x)>0 covers the required support, the importance-sampling identity is:

L*(θ) = − E_{x~q, y~p_r} [ p*(y|x) / p_r(y|x) · log p_θ(y|x) ]. (6)

So far this is standard change of measure. The difficulty is that p* is intractable. The paper defines a large base model p_b^L, small base model p_b^S , and small reference model p_r, assuming:

p*(y|x) / p_b^L(y|x)  ∝  p_r(y|x) / p_b^S(y|x).

Then:

p*(y|x) / p_r(y|x)  ∝  p_b^L(y|x) / p_b^S(y|x),
L_ATL(θ) ∝ −E_{x~q,y~p_r}[ w(x,y) log p_θ(y|x) ],
w(x,y)=p_b^L(y|x)/p_b^S(y|x).                                 (7)

Algorithm 1 on core PDF p.4 also writes the precomputed collection with a term containing p_r(y_i|x_i) , then uses W_i/|B|inside minibatches. This reflects empirical weighting at implementation level, but no public code resolves whether sequence probability means a token product, mean log-probability, or length-normalized score; whether calculations use log space; whether clipping/self-normalization is applied; or how extreme weights are handled. With full-sequence probability ratios,exp(Σ_t Δlog p_t) can vary exponentially with length, allowing numerical stability and length bias to dominate.

Training pseudocode

Inputs: large base p_b^L, small base p_b^S, small instruct model p_r, OpenHermes data D
for (x_i, y_i) in D:
    a_i ← sequence_logprob(p_b^L, y_i | x_i)
    b_i ← sequence_logprob(p_b^S, y_i | x_i)
    c_i ← sequence_logprob(p_r,   y_i | x_i)
    w_i ← exp(a_i − b_i)             # Core proxy in theoretical Equation 7
    # Algorithm 1 also includes p_r; unpublished implementation prevents disambiguation
cache {(x_i,y_i,w_i)}

θ ← parameters(p_b^L)
for minibatch B:
    loss ← − Σ_{i∈B} w_i log p_θ(y_i|x_i) / |B|
    θ ← AdamW(θ, ∇loss)
Output: weighted-SFT model

5. Key mechanisms and engineering details

6. Datasets, benchmarks, and metrics

The paper evaluates knowledge and understanding using lm-evaluation-harnessand preferences using MT-Bench, AlpacaEval, and Arena-Hard-Auto; all metrics are higher-is-better (core PDF p.5 / p.1046, §3.2). There is no blinded human evaluation, out-of-domain deployment traffic, calibration, robustness, or safety regression testing.

7. Complete main-results matrix: five benchmark categories, six method rows

The table transcribes all eight metrics from core Table 2. Parentheses contain absolute differences recalculated as (ATLANTIS−SFT) . Accuracy/win-rate differences are percentage points; MT-Bench single and Arena differences retain their original units. Bold marks the better value within each model.

**Source:** core PDF p.5 / proceedings p.1046, Table 2. Differences and relative percentages below are recalculated here. Llama2's largest relative gain is Arena-Hard, +38.46%; Qwen2's OLL2 gain is +5.44%but Arena changes by −4.04%; Gemma2's MT pairwise gain is +35.40%. Of 24 comparisons, 23 improve and one declines. This supports a broadly consistent direction, not statistical significance.

Gains are heterogeneous: MT single improves by only +0.30、+0.09、+0.30across the three models, versus pairwise gains of +2.50、+0.62、+7.19. The authors argue pairwise judgment is easier and more objective (p.6 / p.1047), but the evaluator may instead favor training-induced style. Without human pairwise validation, genuine helpfulness cannot be distinguished from pleasing the automatic judge.

8. Layered ablations: selection, scale, cross-architecture transfer, and proxy controls

8.1 Combining with IFD / Superfiltering

Table 3 (PDF p.6 / p.1047) varies selection rates of 5%, 10%, and 15%, comparing SFT with ATLANTIS. For Qwen2 with IFD at 10%, ATLANTIS changes TruthfulQA MC1 relative to corresponding SFT by 32.68→35.86（+3.18）、MC2 51.35→54.89（+3.54）、MT single 7.19→7.86（+0.67）、pairwise 24.53→32.81（+8.28）. This is the strongest combination evidence in the paper.

Results are not consistent in every cell. For Gemma2 + IFD 5%, ATLANTIS changes MC1 versus SFT by 31.33→28.76（−2.57）、MC2 49.79→44.24（−5.55）、MT single 3.93→2.73（−1.20）, with only pairwise 6.60→6.88（+0.28）. Superfiltering 5% also yields Gemma2 MC1 33.41→32.93（−0.48）、pairwise 25.00→20.63（−4.37）. Compatibility with selectors is established; universal improvement from combining them is not.

The authors call 10% generally best, but column-wise optima differ. Qwen2 + IFD reaches MT pairwise 32.81at 10%, while Qwen2 + Superfiltering can peak at 15% or ordinary SFT at 10%. Unfiltered Gemma2 ATLANTIS exceeds filtered versions in several TruthfulQA/MT-Bench columns. This resembles task-dependent transitions in the quantity–noise trade-off, not a universal 10% rule.

8.2 Model scale

Table 5 (PDF p.7 / p.1048) reports Qwen2-72B MT pairwise 31.88→36.88（+5.00，+15.68%），TruthfulQA MC1 41.62→42.59（+0.97）and Gemma2-27B MT pairwise 26.56→32.50（+5.94，+22.36%）, but TruthfulQA MC2 61.24→59.30（−1.94）while MC1 stays at 43.82→43.82. Scaling does not remove negative transfer; it changes which benchmark exhibits it.

8.3 Cross-architecture weight transfer

Table 4 (PDF p.7 / p.1048) swaps Qwen2 and Gemma2 small/reference models. Qwen2 using Gemma2 weights achieves TruthfulQA MC1 36.23, above same-family weights at 35.62 and SFT at 35.37, but MT single 6.89、pairwise 21.56remains below same-family ATLANTIS's 7.53 / 26.25. Gemma2 using Qwen2 weights has MT single 7.31 above its same-family value of 6.98，pairwise 25.00 but below 27.50. Figure 4 visualizes 2,000 examples and reports architecture disagreement in positive/negative weighting at threshold 1.0 of 46.78%(PDF p.8 / p.1049). This is not architecture independence: nearly half the examples can reverse direction while some metrics still improve.

8.4 Proxy-tuning control

Appendix Table 7 (PDF p.11 / p.1052) tests only Qwen2-7B. TruthfulQA MC1/MC2 and MT single/pairwise are, for SFT, 35.37/51.45/7.44/25.63; for proxy-tuning, 33.41/50.36/3.76/7.91; and for ATLANTIS, 35.62/52.65/7.53/26.25. Training-time weighting of the shift signal is more effective here than direct decoding combination, but latency, memory, and total FLOPs are not matched. Proxy-tuning avoids large-model training and has a different cost structure.

9. Hyperparameter sensitivity, weight transitions, and statistical significance

Reported sensitivity. Selection rates cover only 5%/10%/15%. Some Qwen metrics peak at 10%, while Gemma2 often prefers no selection. Scaling from 7→72B and 9→27B shows gains need not grow monotonically. Cross-architecture weighting shows 46.78% directional disagreement. These constitute discrete sensitivity analyses.

Important unreported sensitivity. No grids cover weight temperature w^τ, clipping caps, self-normalization, sequence-length normalization, reference-training strength, small-model size, or corpus source. Effective sample size ESS=(Σw)^2/Σw², weight quantiles, and gradient norms are also absent. It is unclear whether smooth reweighting or a few extreme examples drive updates. The characteristic variance problem of importance sampling is unmeasured.

Statistical significance. The title, abstract, and text describe consistent and significant improvements, but Appendix Table 6 states that every experiment uses only seed 0. Main tables provide no standard deviations, confidence intervals, bootstrap, or paired tests. This report can therefore state only that 23/24 comparisons improve, not that improvements are statistically significant. On finite-sample TruthfulQA or MT/Arena judging, +0.09 or +0.10 may fall within noise.

10. Contributions, reusable designs, and transfer potential

The substantive contribution is not the label of first using importance sampling, but a convenient post-training combination of three existing ideas. Proxy-tuning suppliestransferable small-model shifts; weak-to-strong selection suppliessmall models as proxies for evaluating examples; weighted SFT supplies an implementation thatleaves model architecture and the training loop unchanged. Once three-model probabilities are cached offline, the trainer reads one scalar weight per example.

Three designs are reusable. First, define example value throughchanges in model staterather than an external absolute quality score, capturing family-specific utility. Second, combine continuous weights with hard selection: cheaply remove redundant examples using a small model, then refine the remainder. Third, partial cross-architecture transfer suggests shared score stores, but 46.78% directional disagreement must be treated as a version-compatibility metric.

For code, medical, or multilingual training, a sensible approach is to trainsmall in-domain reference modelsrather than reuse a general chat reference, and explicitly relate density ratios to deployment queries while recording ESS and length correlations. Continual learning could use old/new small-model shifts for each arriving batch. Preference optimization could use the ratio as an example prior, ablated separately from DPO margins. Private large models are difficult targets because training and scoring require large-base weights or at least token-level log-probabilities.

11. Technical lineage of related work

11.1 Weak-to-Strong Generalization: the question, with a different supervision format

Problem addressed: Burns et al. study weak-model labels supervising stronger models and observe students outperforming weak teachers on GPT-4-family NLP, chess, and reward modeling. The abstract also reports that an auxiliary confidence loss brings GPT-4 under GPT-2-level supervision toward GPT-3.5-level performance (W2S PDF p.1).Remaining gap: Weak-label supervision does not directly determine how much to weight each open-ended instruction response.ATLANTIS's response: Turn the weak teacher from a label generator into a probability-shift probe.Transferred design: The weak model is a probe for transferable training signals, not the final capability ceiling. W2S does not prove ATLANTIS's density-ratio identity.

11.2 Proxy-tuning: the direct origin of probability shifts

Problem addressed: At decoding, proxy-tuning adds small tuned-minus-base logits to the large base model, without updating it. Its abstract reports that a 7B proxy guiding Llama2-70B closes 88%（Proxy PDF p.1）。Remaining gap: Multiple models run for every token, increasing inference cost. Core Appendix Table 7 also finds much worse Qwen2-7B MT-Bench results than SFT.ATLANTIS's response: Move the same shift signal to pretraining-time example weighting, eliminating multimodel decoding at deployment.Transferred design: Approximate additive/proportional transfer across model scales enters ATLANTIS Equation 7 almost unchanged and is its most important assumption to validate.

11.3 IFD: defining learning value through conditional answer probabilities

Problem addressed: IFD compares answer perplexity with and without instructions to select examples that teach instruction following. Its abstract reports improved Alpaca/WizardLM training using only 10% of the original data (IFD PDF p.1).Remaining gap: Scoring with the target model is costly, and hard top-k selection loses coverage.ATLANTIS's response: Construct continuous weights from small/reference-model probabilities while retaining all data.Transferred design: Both define value through conditional-probability changes rather than static semantic quality. ATLANTIS Table 3 directly uses IFD as a compatible preliminary selector.

11.4 Superfiltering: evidence that small-model scores transfer

Problem addressed: Superfiltering computes IFD using a GPT-2-scale model. Figure 1 / the abstract reports selection speedup of about 20×and shows that 5% of data can outperform training on 100% (Superfiltering PDF p.1).Remaining gap: It remains hard selection with a tunable fraction, and changing model families may change rankings.ATLANTIS's response: Move from ranking to influence weights and test cross-architecture transfer.Transferred design: Weak models can proxy data value, but ATLANTIS's 46.78% cross-architecture directional disagreement warns that this transfer is fragile.

11.5 LESS: data influence guided by optimization trajectories

Problem addressed: LESS constructs a low-rank gradient datastore accounting for Adam and variable-length sequences, then selects training data using a few target-task examples. Its abstract reports that selecting 5% often exceeds full-data training (LESS PDF p.1, ICML 2024).Remaining gap: Gradient features must be stored/retrieved, and target-task examples are needed.ATLANTIS's response: Use only forward probabilities, without target gradients or downstream few-shot examples.Transferred design: Data value should reflect where optimization moves the model. LESS measures this explicitly through gradient inner products; ATLANTIS approximates it with base-to-tuned probability changes.

Nonlinear technical evolution

                                                      ┌─ Weak-to-Strong supervision
                           │   Can weak labels unlock a strong student?
                           │                 │
Full-data SFT → Data-value modeling ─┼─ IFD ─→ Superfiltering
  p_d ≠ deployment distribution     │  Self-difficulty   Transferable small-model scores
                           │        │          │
                           │        └────┬─────┘
                           │             ▼
                           ├─ LESS: gradient / trajectory influence ─┐
                           │                                         │
                           └─ Proxy-tuning: logit shifts ────────────┼─→ ATLANTIS
                                                                     │   Probability shifts × continuous SFT weights
Importance sampling: change-of-measure identity ─────────────────────┘
                                                                     │
                       Unresolved: p* definition, p_r/data mismatch, heavy tails, statistical tests

Two shifts drive the evolution: from more instructions to identifying which instructions change the model; then from costly large-model/gradient scoring to transferable weak-model proxies. ATLANTIS connects the second shift to importance-sampling terminology, but target distributions and support conditions do not yet complete its theoretical justification.

Cross-paper comparison

12. Limitations acknowledged by the paper

1. Dependence on a smaller same-family model. The authors recommend smaller models from the same family to realize ATLANTIS's benefits. Cross-architecture influence-weight distributions affect performance and limit applicability (core PDF p.9 / p.1050, Limitations). Table 4 and 46.78% disagreement provide quantitative support.
2. Additional computation. Reference models increase cost relative to vanilla SFT; the authors deem this generally acceptable because the references are small (same page). Without cost figures, that assessment cannot be audited.

The paper gives no third independent limitation, so none is invented. The following critiques are this report's own.

13. Independent critique: evidence → implications

1. [Theory / unidentifiable target] Because p* is defined as an optimal distribution covering all real-world knowledge and expression, without observable samples, a target population, or a utility function (pp.1–3). Finite-benchmark gains therefore cannot verify movement toward p*; the symbol functions more as an idealized narrative target.
2. [Theory / broken change of measure] Exact importance sampling requires samples from proposal distribution p_r, whereas experiments use OpenHermes-2.5 andp_r is an official chat/instruct model (p.5, Table 1). Equation 6 is therefore not equivalent to actual minibatch training. Without an additional data-density ratio, the method is more accurately proxy-probability-weighted SFT.
3. [Numerics / heavy tails] Sequence-probability ratios can exponentially accumulate small token differences on long responses. Without log-space implementation details, length normalization, clipping, or ESS, it is unclear whether stable distribution correction or extreme examples drive improvement.
4. [Statistics / overclaiming] All experiments use seed 0 (Appendix Table 6), without deviations or tests. Significant therefore lacks statistical support, especially for +0.09 Qwen MT single and +0.10 Qwen Alpaca hard, which cannot be distinguished from noise.
5. [Empirics / inconsistent gains] Main-table Qwen Arena 9.9→9.5，Gemma2‑27B TQA MC2 61.24→59.30and several substantial Gemma2+IFD 5% declines mean steady across models can describe an overall trend, not a per-task safety guarantee.
6. [Attribution / training-data leakage] Official instruct/chat training data are not fully disclosed, and OpenHermes is a broad synthetic mixture. High weights may mark reference-model familiarity or preferred style rather than proximity to a real-world optimum. Deduplication/contamination audits are needed to distinguish these explanations.
7. [Cost / incomplete accounting] Training ≤13B models already uses 16×A800 and 72B uses 64×A800, plus three-model forward precomputation. Without GPU-hours, tokens/s, or storage costs, small reference models do not establish acceptable end-to-end overhead.
8. [Evaluation / judge preferences] Largest gains concentrate in MT pairwise, up to +7.19, while MT single improves only +0.09–0.30. Both use automatic judges, so comparison-friendly response style may explain much of the gain. Human-preference improvement lacks blinded human evidence.

14. Open challenges: actionable head-to-head experiments

Theory

• True density ratios versus the ATLANTIS proxy. Construct known distributions on an enumerable synthetic language, p*、p_d、p_rthen compare true ratios, ATLANTIS, random weights, IFD, and uniform SFT at equal FLOPs. Report KL to p*, ESS, and gradient cosine similarity to directly test proportional-transfer consistency.
• Heavy-tail stabilization. Fix data and checkpoints; compare raw ratios, log-ratio z-scores, self-normalized IS, clipping at the 95/99/99.9th percentiles, and Pareto-smoothed weights. Across five seeds, report means, confidence intervals, ESS, and worst-task performance to identify performance–variance transitions.
• Sequence-probability definitions. Compare token products, mean log-probabilities, length-bucket normalization, response-only scoring, and instruction+response scoring. Plot Spearman weight–length correlations to test length-bias explanations.

Engineering and applications

• Equal-GPU-hour cost curves. Under the same 8×H100 budget, compare more vanilla-SFT steps, ATLANTIS three-model scoring plus training, LESS datastore construction, 10% Superfiltering, and proxy-tuning. Plot end-to-end GPU-hours/disk use against five-benchmark macro averages and p95 latency.
• Cross-domain references. For code, medicine, and multilingual tasks, compare general-chat and in-domain tuned small references with the same large base and token budget. Measure in-domain gains, out-of-domain forgetting, and safety regressions. Domain-only gains at the expense of general capability would challenge a unified p* account.
• Contamination controls. Build dynamic QA and private-template sets after the training cutoff, with strict MinHash/embedding deduplication. Compare public instruct references with references trained on nonoverlapping data to test whether weights mainly detect memorization.

Emerging directions

• Combining with DPO/GRPO. On identical preference pairs, compare DPO, ATLANTIS-weighted SFT→DPO, ATLANTIS-weighted DPO, and margin-based weighting. Report reward, KL, length, hacking rate, and stability to separate sample priors from preference objectives.
• Online continual weighting. Freeze old/new small references after each data arrival and compare static with time-conditioned weights. Measure knowledge acquisition, forgetting, and weight drift on knowledge-update benchmarks to test lightweight continual post-training.
• Cross-architecture compatibility gates. Before training, score only 2,000 examples with both weighting schemes. If directional disagreement exceeds a threshold, fall back to same-family models. Compare against ungated ATLANTIS on worst-task performance, overhead, and calibration.

15. Recent-work radar

• Model Extrapolation Expedites Alignment（ACL 2025 Main）: also targets cheaper alignment training, but through parameter extrapolation; compare post-training with ATLANTIS at equal total FLOPs.
• Intuitive Fine‑Tuning（ACL 2025 Main）: decomposes SFT and preference optimization into preference estimation and transition optimization, then unifies them through temporal residuals—a more executable target than a real-world optimal distribution.
• Packing Analysis（ACL 2025 Findings）: studies SFT packing benefits across model/data scales, an infrastructure variable that matters when evaluating ATLANTIS's end-to-end efficiency.
• Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding: applies weak-to-strong transfer to low-resource preference decoding, helping distinguish costs of training-time weights from inference-time guidance.
• DenoiseRL（arXiv:2605.28421）: recovers noisy reasoning supervision through RL, suggesting weak signals need explicit noise modeling as well as weighting.
• LESS（ICML 2024）: optimizer-aware target-data selection remains a strong interpretable alternative. Without weight–gradient agreement, ATLANTIS cannot establish that probability ratios outperform direct optimization influence.
• Superfiltering（arXiv v2）: directly precedes transferable weak-model scoring and highlights selection fraction and model family as independent hyperparameters.

16. Final assessment and reproduction priorities

ATLANTIS compresses an expensive data-value problem into forward probabilities from three existing models and supplies broad experiments across five benchmark categories, three families, larger scales, and cross-architecture settings. Its strongest empirical result is 23/24 improving comparisons in the main table, including Gemma2-9B MT-Bench pairwise 20.31→27.50（+7.19，+35.40%）. Counterevidence includes Qwen Arena regression, substantial negative transfer in some Gemma2 selection combinations, and a single seed without significance testing.

The first reproduction priority is publishing weight computation and plotting weight quantiles, length correlations, ESS, minibatch gradient norms, and five-seed results. Next, separate OpenHermes's empirical distribution p_d from reference model p_r and incorporate an estimable data-source density ratio. Only then can importance sampling toward a real-world optimum become a testable method rather than an attractive interpretation. Until then, the best-supported description is:An effective continuous example-weighting heuristic based on small-model instruction-tuning shifts, with incomplete theoretical justification.

Evidence and generation notes: All performance numbers come from core ACL Tables 2–7 or the five related PDFs read, with page/table references. Percentages are recalculated from original values, never inferred from pixels. Core code, error bars, weight-stabilization details, and complete costs are unavailable, and corresponding claims remain marked not reported / not inferable.ml-report-index.md was updated. HTML was generated from the Warm Editorial template and passed its validator with 0 warnings. Playwright loads, but /usr/bin/chromium、/usr/bin/chromium-browser、/opt/pw-browsers and its cache contain no executable Chromium, soreal light/dark/mobile screenshot checks remain incomplete. Static responsive checks are not presented as browser validation.