Generalization and Robustness of the Tilted Empirical Risk

0. Verified links and evidence conventions

Links accessed 2026-09-19. Exact core numbers come from arXiv v3 HTML tables/text. Recomputations use arithmetic on reported values, never plot estimates disguised as exact figures.

Page/line conventions. Core definitions/theorems occupy paper pp.2–5, arXiv HTML lines 98–205; data-driven selection/main results pp.6–7, lines 300–354; appendix experiments lines 1028–1079. Because HTML lacks stable printed pagination, citations combine table/theorem/equation numbers with HTML lines and PDF ranges where needed. Recomputed differences use the same reported row values.

1. Candidate pool, deduplication, and random selection

First checked ml-report-index.mdand excluded recently covered trust/theory themes: Emergent Misalignment (09-05), diffusion memory (09-09), coordinate-missingness robust regression (09-12), and multimodal safety alignment (09-15). Candidates were cross-collected through ICML/PMLR, OpenReview, arXiv, ACL Anthology, Hugging Face Daily Papers, Papers with Code, and Semantic Scholar. Some auxiliary sources supplied only aggregations or unstable links; identities follow official proceedings/OpenReview/arXiv.

The frozen 24-paper pool was: ① Enhancing Adversarial Robustness with Conformal Prediction; ② What Makes an Ensemble Interpretable?; ③ Explaining Fast and Slow; ④ Differentially Private MCMC; ⑤ GaussMark; ⑥ Probably Approximately Global Robustness Certification; ⑦ TRUST-VLM; ⑧ Exactly Tight Information-theoretic Generalization Bounds; ⑨ Improving Zero-Shot Adversarial Robustness of VLMs; ⑩ A Lens into Interpretable Transformer Mistakes; ⑪ ERICT; ⑫ Active Reward Modeling; ⑬ Privacy-Shielded Image Compression; ⑭ Weak-to-Strong Jailbreaking; ⑮ Stable Fair Graph Representation Learning; ⑯ Avoiding Leakage and Poisoning; ⑰ Generalization and Robustness of the Tilted Empirical Risk；⑱ Prediction via Shapley Value Regression；⑲ Mechanistic Interpretability to Biology；⑳ Speak Easy；㉑ De-mark；㉒ Subgroups Matter；㉓ False Coverage Proportion Control；㉔ Beyond Black-Box Agentic Tool Use。

Command executed: shuf -i 1-24 -n 1returned 17. The ICML 2025 paper was complete and within 24 months, so redraw count was 0. The full pool and discovery entries are preserved in the accompanying candidate record.

2. Motivation: where ERM breaks

Standard empirical risk minimization uses average loss

$$\widehat R(h,S)=\frac1n\sum_{i=1}^n \ell(h,Z_i),$$

Linear, unbiased, and familiar to optimize, it also lets extreme losses enter with equal coefficients. Heavy tails may lack variance; a few outliers can pull fitted parameters. With 44.45% Pareto outliers in linear regression, ERM population risk is 2.2190±0.0011versus data-driven negative tilting 0.4908±0.0001(Appendix Table 4, lines 1059–1066), an absolute reduction 1.7282 and relative reduction 77.88%. At 44.45% Gaussian outliers, ERM is 0.3070±0.0001versus negative tilting 0.15399±0.0001(Appendix Table 5, lines 1067–1074), reducing 0.15301 absolutely and 49.84%。

Robustness costs something: stronger negative tilting drives extreme-loss exponential weights toward zero, reducing contamination influence but also suppressing legitimate difficult examples. The theoretical $e^{|\gamma|\kappa}$ term grows. As $\gamma\to0^-$, TER returns to ERM, removing both tilt bias and protection. The problem balances three quantities:

1. Tilt bias: TER no longer directly estimates mean population risk;
2. Finite-sample estimation error: exponentiation followed by logarithms has stability depending on $n,\gamma$ and moment assumptions;
3. Training–test distribution mismatch: heavier contamination calls for more negative $\gamma$ to suppress outliers.

The question is whether unbounded losses with only a finite $(1+\epsilon)$ -order moment permit uniform-convergence, information-theoretic, and shift guarantees for negative TER, enabling selection of $\gamma$. This goes beyond original TERM's one-knob unification of fairness, hard-example emphasis, and outlier suppression to ask when that knob has statistical guarantees.

3. Argument, core assumptions, and trade-offs

Four steps structure the paper: define tilted generalization error as population risk minus TER; split same-distribution error into mean-versus-entropic tilt bias and population-versus-empirical entropic sampling error; introduce total variation between training distribution $\tilde\mu$ and test distribution $\mu$ as a robustness term; combine bounds into an optimizable objective for $\gamma$ and extend to KL-regularized randomized learners.

The crucial Assumption 3.1 states that for every $h\in\mathcal H$there exists $\kappa_u>0$such that

$$\mathbb E_{\mu}[\ell(h,Z)^{1+\epsilon}]\le \kappa_u^{1+\epsilon},\quad \epsilon\in(0,1].$$

This is weaker than bounded loss and permits nonexistent second moments—for example Pareto shape 1.5 with $\epsilon=0.5$—but remains auniform moment bound over the entire hypothesis space. The main uniform theorem first treats finite $\mathcal H$; its unbounded-loss lower bound requires $n$ above a threshold involving $e^{2\gamma\kappa_u}$ (Theorem 3.5, lines 189–199). Unbounded is not unconditional: tails need the stated moment, sample size must exceed a threshold, and hypothesis-space size enters through $\log|\mathcal H|$ .

The central tradeoff is:$|\gamma|^\epsilon\kappa_u^{1+\epsilon}$ grows with tilt magnitude, representing mean-versus-entropic bias; contamination scales roughly with TV times a $\gamma$ -dependent exponential difference quotient and can decrease under stronger negative tilt; estimation also contains $1/|\gamma|$、$e^{|\gamma|\kappa_u}$ and $n^{-1/2}$. The knob is nonmonotonic, yielding U-shaped or more complex transition landscapes.

4. Equation-level formalization

5. Executable workflow and engineering details

Input: training set S, clean reference S_ref if available, candidate γ<0, model hθ
1. Estimate ε and moment scale κ_s; if S_ref exists, estimate κ_u and contamination-TV bound
2. For each γ, compute B(γ)=tilt bias+contamination term (finite-sample terms should also be included in practice)
3. Select γ_data = argmin B(γ)
4. Optimize iteratively:
      loss_i ← ℓ(hθ, z_i)
      w_i ← softmax(γ loss_i)
      θ ← θ - η Σ_i w_i ∇θ loss_i
5. On clean validation data unused for γ selection, report mean/tail risk and calibration
Output: hθ and γ_data; record effective sample size 1/Σ_i w_i²

Actual implementation must be distinguished from stronger engineering recommendations. Reported settings below repeat verifiable configurations; recommendations are proposed transfer safeguards.

6. Datasets, benchmarks, and metrics

No public real dataset is used. Evaluation covers two tasks, two outlier distributions, and five contamination levels: Logistic/Gaussian, Logistic/Pareto, Linear/Gaussian, Linear/Pareto, plus Pareto Logistic without explicit outliers. The metric is population risk under clean distribution $\mu$ , lower is better—not accuracy, AUC, or training TER.$\rho$ is outlier count relative to original training size. Linear levels 0.049%, 9.13%, 16.70%, 33.35%, 44.45% differ from Logistic levels 0.1%, about 17.6%, 35%, 52.5%, 70%.

Only three methods exist: oracle-grid TERM, ERM, and data-driven TERM. A five-benchmark×six-method matrix cannot be honestly manufactured. Tables show four comparable scenarios×five contamination levels×three methods, with means/std from Tables 1,2,4,5. Parenthesized relative-risk changes use ERM as baseline. Where ERM/oracle rounds to zero, ratios are undefined; only absolute differences or undefined labels appear.

7. Complete main-results matrices

8. Ablations, sensitivity, and transitions

9. Genuine contributions, reusable design, and transfer potential

First, TER is anchored back to ordinary population mean risk rather than its own entropic objective; Proposition 3.2 quantifies mismatch, avoiding winning an old problem through a new metric. Second, only a finite $(1+\epsilon)$ -order moment is needed for unbounded losses, with uniform and mutual-information bounds connecting TERM to heavy-tailed statistics beyond bounded loss. Third, an explicit TV shift term enters $\gamma$ selection as an optimizable bias–robustness proxy. Fourth, tilted Gibbs posteriors connect log-sum-exp weighting with KL regularization/SKL information, supporting Bayesian/randomized optimization transfer.

Four reusable designs follow:

1. Separate objective mismatch before concentration. Nonlinear risks need a bound against the final evaluation risk, not merely surrogate self-consistency.
2. Select robustness hyperparameters by interpretable bounds. Even loose constants expose unknown quantities and guide data collection.
3. Report both oracle and deployable selectors. Their gap is itself a research target; oracle performance is not method performance.
4. Monitor effective sample size. For exponential reweighting,$1/\sum_iw_i^2$ reveals whether strong negative tilt trains on very few low-loss points. Unreported here, it is essential for transfer.

Off-policy learning already applies log-sum-exp to heavy-tailed importance-weighted rewards. Federated learning can tilt client losses; LLM post-training can downweight anomalous preference pairs; time series can suppress bursts. Each must distinguish bad data from rare but important hard examples. Minority groups may be intrinsically harder, so average robustness can harm worst-group fairness.

10. Related work: origins, gaps, and design transfer

11. Nonlinear technical evolution and turning points

                         Ordinary ERM: mean loss
                        /                \\
        Risk-sensitive learning: entropic risk   Robust statistics: truncation / Catoni / MOM
                    |                         |
                    |                         +-- finite moments, different objectives/optimization
                    v
        TERM / Tilted Losses (2021→2023)
          ├─ γ<0: soft-min, outlier downweighting
          ├─ γ=0: ERM
          └─ γ>0: soft-max, difficult groups/fairness
                    |
          ┌─────────┴──────────┐
          v                    v
  Information bounds / Gibbs   LSE Off-policy (ICML 2025)
          |                    |
          └─────────┬──────────┘
                    v
  Generalization & Robustness of TER (ICML 2025)
          ├─ unbounded loss + (1+ε)-order moments
          ├─ uniform / MI generalization bounds
          ├─ TV contamination and γ_data
          └─ tilted Gibbs posterior
                    |
          ┌─────────┼────────────────┐
          v         v                v
   tight divergences real/deep models joint estimation without clean references
   (Binary-JS)       and serving cost (TV, ε, κ, γ)

Drivers and turning points. First, means become adjustable tail objectives, with log-sum-exp unifying soft-min/max. Second, an optimization knob becomes a statistical object requiring ordinary population-risk control, not only stable training. Third, same-distribution generalization expands to contamination:$\gamma$ depends on shift as well as sample size. The unfinished fourth shift moves from proofs knowing $\mu$, TV, and tail indices to deployment with only one mixed training set.

12. Limitations acknowledged by the paper

1. Overparameterization is absent. Future Works (§9, lines 412–416) explicitly leaves overparameterized/mean-field regimes open. The main uniform expression first depends on finite $|\mathcal H|$, so it cannot directly cover modern deep networks.
2. Unbounded losses still require sample thresholds. Section 9, line 417 acknowledges a lower bound on $n$ ; only the bounded-loss appendix relaxes it. Because thresholds involve $\gamma,\kappa,\delta,\zeta$, a theoretically selected $\gamma$may lie outside theorem applicability at small sample sizes.
3. Positive tilting is unresolved. Section 9 lines 418–420 and Impact Statement lines 424–426 explicitly analyze only $\gamma<0$for unbounded losses, leaving positive-tilt imbalance/fairness uses open. Core guarantees cover only half the TERM knob.

13. Independent critique: evidence → implications

1. [External validity] Because exact experiments use only 2D synthetic Logistic/Linear data with 1,000/2,000 samples and no real/deep models, up to 77.88% relative risk reduction cannot be extrapolated to real systems; it establishes the mechanism in controlled families.
2. [Deployable tuning] Because $\gamma_{data}$ is implemented using true data to estimate $\kappa_u$and known outlier counts for TV bounds (lines 1028–1036). Data-driven still includes normally invisible clean-distribution/contamination information: it is a semi-oracle selector, not fully unsupervised selection.
3. [Oracle attribution] Because $\gamma^\star$ grid-searches clean population risk directly (lines 322–336), making it a potential upper reference, not deployable baseline. Claiming its 2.2190→0.1229, 94.46% reduction as method performance leaks test information.
4. [Theoretical tightness] Because bounds contain $e^{|\gamma|\kappa}$、$1/|\gamma|$, TV, and unknown moment constants without bound/value ratios or coverage, asymptotic structure is established but nonvacuity at experimental scale is not.
5. [Statistical reliability] Because cells use only three runs without CIs/tests, even tiny standard deviations cannot characterize rare Pareto extremes. Three-run mean superiority is not statistical significance.
6. [Fairness side effects] Negative tilting downweights loss rather than known anomalies, also suppressing minorities, tail classes, and legitimate hard examples. Mean robustness may cost worst-group risk; subgroup metrics are absent.
7. [Formal consistency] Because the appendix says squared loss but writes $\ell(h,z)=\tfrac12(h^Tx-y)$without the square (line 1049), there is a typesetting/implementation-description error. Without code, the actual loss in Tables 4–5 cannot be confirmed.
8. [Selector mismatch] At 44.45% Linear/Pareto contamination, data-driven risk 0.4908 is 299.35% above oracle 0.1229, versus only 0.65% for Linear/Gaussian. The proxy is highly tail-family-sensitive; unknown $\epsilon$ or moment-estimation error may dominate practical failures.

14. Open challenges: actionable head-to-head experiments

15. Recent-work radar (verified links)

1. Log-Sum-Exponential Estimator for Off-Policy Evaluation and Learning（ICML 2025）: the same operator yields $O(n^{-\epsilon/(1+\epsilon)})$ regret for importance-weighted rewards, extending beyond supervised regression, though propensities/bandit structure introduce different biases.
2. Exactly Tight Information-theoretic Generalization Bounds via Binary Jensen–Shannon Divergence（ICML 2025）: directly addresses loose information bounds and provides a next-generation comparator for the MI branch; availableCode。
3. Adaptive Estimation and Learning under Temporal Distribution Shift（ICML 2025）: extends static TV contamination to temporal shift with sharp final-time rates, providing online $\gamma_t$ with a more realistic drift model.
4. Model Assessment and Selection under Temporal Distribution Shift（ICML 2024）: adaptive rolling windows/tournaments inspire selection of $\gamma$without clean-distribution knowledge, though they select models rather than reweight examples.
5. A Systematic Review on Long-Tailed Learning（arXiv 2024）: emphasizes that long-tailed classes differ from heavy-tailed losses/outliers. Visual TERM transfer must report class fairness rather than suppress all high-loss tail examples.
6. Generalization and Robustness of the Tilted Empirical Risk（ICML 2025）: this report's core paper; its strength is unified bounds and explicit $\gamma$ tradeoffs, with deployment observability and empirical scale as weaknesses.

16. Final assessment

The paper does not establish negative $\gamma$as the default for robust learning. It clarifies where ordinary risk, exponential tilt bias, finite-sample concentration, contamination, and algorithmic information enter TERM guarantees. For theorists, combining$O(n^{-\epsilon/(1+\epsilon)})$ with TV shift is a baseline to tighten. For practitioners, the oracle–data-driven gap reveals tail indices, contamination rates, and clean references as deployment bottlenecks.

The key result: at 44.45% Linear/Pareto contamination, data-driven TERM reduces ERM risk 2.2190→0.4908 (−77.88%) yet remains 299.35% above oracle 0.1229; under Linear/Gaussian it is only 0.65% above oracle. Evidence supports strong robustness potential, not a selector already reliable across tail distributions.

Production and audit footnote. URLs were actually accessed; absent code/data/weights/project pages are marked. The environment lacks an executable browser at /usr/bin/chromium or /opt/pw-browsers , and, as required, did not run playwright install. Warm Editorial templates, structural validation, static responsiveness, and semantic checks substitute for screenshots without claiming browser rendering.ml-report-index.md was updated. No unreported real-data results, significance, exact sensitivity points, or deviations are invented.