HPS: Turning Hard Negatives into a Preference-Alignment Sampling Distribution

Link verification

Unverified items. PMLR's Software link confirms the GitHub destination, but direct retrieval timed out, so latest commit, license, and added weights were not independently checked. OpenReview's destination was confirmed but content blocked. Links remain without claiming full reproducibility.

Rotation, deduplication, candidate pool, and random draw

date +%u returned 1, selecting Monday's LLM/NLP category. First checked ml-report-index.mdand excluded recently covered ATLANTIS(SFT data weighting, 2026-09-07) and ThinkSLM(small-model reasoning evaluation, 2026-09-14), also avoiding MM-RLHF (09-15). ICML 2025 PMLR supplied the pool, cross-searched through arXiv, OpenReview, Hugging Face Papers, Semantic Scholar, and Papers with Code; official proceedings resolve missing aggregator pages.

After freezing candidates, executed shuf -i 1-24 -n 1returned 24, selecting HPS. Paper, final version, tables, code entry, and two datasets provided sufficient material, sono redraw was needed. Random selection determines the core paper but does not replace evidence requirements; this draw meets the recent-main-conference priority.

Narrative, assumptions, and tradeoffs

Four steps summarize the approach:

1. Redefine informational value. Rejected answers closest to preferred ones are hardest to distinguish and potentially most informative.
2. Turn rankings into distributions. Fixed estimated rewards $r_{est}$ pass through temperature $gamma$ softmax to form $q$；$gamma$ ; larger values approach hardest-negative sampling.
3. Sample one negative. Rather than backpropagating through every candidate, draw from $q$ and pair it with the top response for ordinary pairwise optimization.
4. Explain the limit. When $gamma\to\infty$Sampling approaches worst-case negatives, and the limiting objective maximizes the minimum margin against the strongest rejected answer (Theorems 2–3, lines 207–227).

Assumptions span three areas: bounded implicit rewards, gradients, Hessians, and optimal parameters within radius $B$ (Assumption 1, lines 187–192); ranking by $r_{est}$ must align with true hardness; safety assumes every candidate in $j\ge2$ should be downweighted. Real preferences include equally safe styles, weak/nontransitive orders, and conflicting objectives; rejected is not synonymous with harmful.

The tradeoff is clear: HPS exchangesoffline candidate generation and external labeling costfor lowertraining-time backpropagation cost. Only one negative is backpropagated, but generating 100 answers and labeling on 8×L40S remains expensive; training uses 4×L40S (Appendix D.1, lines 732–736). The 61.76% reduction concerns fine-tuning duration, not total data-production-plus-training cost.

Equation-level formalization and workflow

For prompt $x$ and ranked candidates $y_{\tau(1)}\succ\cdots\succ y_{\tau(n)}$, DPO-style implicit reward is

$$r_\theta(x,y)=\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{ref}(y\mid x)}.$$

HPS defines a hardness distribution over rejected candidates (Eq.9, lines 173–178):

$$q(x,y)=\frac{\exp(\gamma r_{est}(x,y))}{\sum_{i=2}^{n}\exp(\gamma r_{est}(x,y_{\tau(i)}))},\qquad y\in\{y_{\tau(2)},\ldots,y_{\tau(n)}\}.$$

Substitute it into the preferred-versus-weighted-negatives objective (Eq.10):

$$\mathcal L_{HPS}=\mathbb E_{d\sim\mathcal D}-\log\frac{e^{r_\theta(x,y_{\tau(1)})}}{e^{r_\theta(x,y_{\tau(1)})}+N\,\mathbb E_{y\sim p(y)}[e^{r_\theta(x,y)}q(x,y)]},\quad N=n-1.$$

The implementation does not calculate the inner expectation exactly, instead using $y^-\sim q$ single-sample Monte Carlo, retaining a DPO-like paired update. Theorem 3 gives the limiting objective

$$\theta^*=\arg\max_\theta\left[r_\theta(x,y_{\tau(1)})-\max_{1<j\le|\tau|}r_\theta(x,y_{\tau(j)})\right],$$

as maximin margin (Eq.14, lines 224–227). Larger internal margins do not automatically lower real harmfulness; calibrated independent safety measures without shared model biases are missing.

for each prompt x:
    generate / retrieve n candidate responses
    keep y+ = rank-1 response
    score every rejected y_i with frozen r_est
    gamma <- linear schedule(-5, -3, -1, 1, 3, 5) at 20% intervals
    q_i <- softmax(gamma * r_est_i)
    sample one y- ~ q
    update chosen preference optimizer on (x, y+, y-)

Note the method states$gamma>1$, while experiments increase from −5 to 5 linearly(§4.2 line 176; implementation line 315). Negative $gamma$ preferentially sampleslow-reward, easy-to-rejectanswers, opposite hard-negative sampling. This can be read as an easy-to-hard curriculum, but the transition is not integrated into the definition/theory.

Key mechanisms and engineering details

Datasets, benchmarks, and metrics

HH-RLHF contains multi-turn helpful/harmless dialogues; the public expanded dataset card used by the authors lists train 161k rows, with a response sequence length of 3 per entry (dataset card lines 72–80).PKU-SafeRLHF / PKU-Safety is a single-turn safety preference dataset. Each prompt originally has two responses in this paper, subsequently expanded to 100. Thus, “sample count” includes both the number of prompts and candidates per prompt; 100 times as many candidates must not be treated as 100 times as many independent human annotations.

Four metric types: BLEU measures lexical overlap with reference answers and is only a weak proxy for open-ended dialogue; Reward comes from external ArmoRM-Llama3; RM_DPO is the difference in implicit log-ratios between preferred and rejected responses; RM_R-DPO additionally subtracts $0.01(|y^+|-|y^-|)$ to correct for length (Table 1, HTML lines 228–232). Human evaluation samples 15 prompts from each of the two test sets, totaling 30 questions and 4 system responses, rated by20 raters on a 1–5 Likert scale, anonymously and in randomized order (Section 6.1, lines 321–323). Qwen2.5-72B-Instruct provides 0–5 scores and pairwise win rates.

Complete main-results matrix

Table 2 covers fine-tuning. For completeness, the table below lists SFT and 5 optimizers×3 ranking models, totaling 16 rows×8 numerical columns; bold indicates only the maximum in each column, without treating implicit margins across incomparable optimizers as an absolute safety scale. Source: Table 2, arXiv HTML lines 233–254.

The matrix shows both a consistent direction and exceptions. On HH, HPS generally improves both margins over the corresponding BT/PL variants for each optimizer; external Reward, however, does not systematically improve: DPO falls by 0.001, and IPO drops from BT’s 0.428 to 0.424. Under PKU fine-tuning, EXO-HPS Reward of 0.425 and SPPO-HPS of 0.435 improve over their respective families, but DPO-HPS Reward of 0.407 is below DPO-BT’s 0.417, an absolute −0.010 or relative −2.40%. A “larger margin” therefore cannot be summarized as “better on every external dimension.”

Human evaluation. Table 3 quality scores: SFT 3.63, DPO-BT 3.82, DPO-PL 3.69, DPO-HPS 3.93. HPS gains +0.11 (+2.88%) over BT, +0.24 (+6.50%) over PL, and +0.30 (+8.26%) over SFT; the paper provides no standard deviations or significance tests. Table 4 Qwen judge: HPS win rates against SFT / BT / PL on HH are 63.875/58.712/57.612%, and on PKU are 67.100/56.100/57.650%; corresponding loss rates reach as low as 5.150%. These are meaningful external evidence, but the judge and training annotators belong to the same public LLM/RM ecosystem and may still share biases.

Transfer setting. In Table 5, DPO-HPS has PKU RM_DPO of 5.725versus DPO-BT’s 1.346, an absolute +4.379 or relative +325.33%; RM_R-DPO is 5.116 vs 1.738, +3.378 or +194.36%. Yet both have Reward of 0.407, and BLEU is only 0.310 vs 0.308. In other words, the transfer experiment reinforces a “steeper internal boundary” without showing a corresponding rise in external reward.

Layered ablations, hyperparameter sensitivity, and phase transitions

Candidate count. In the HH ablation of Table 6, DPO-HPS RM_DPO changes from $n=5$ interval's 0.600 without JVP to $n=100$ interval's 2.723, +2.123 or relative +353.83%; DPO-BT rises from 0.166 to 0.349, +0.183 or +110.24%. BLEU and Reward remain essentially flat. Expanding the support set mainly changes the rejection boundary, rather than the preferred answer’s wording or external RM quality. This also constitutes a phase transition: larger candidate pools are more likely to contain “high-scoring but incorrect/harmful” hard negatives, and HPS benefits grow with extreme-value statistics.

Transfer ablation. Appendix Table 7 gives PKU HPS RM_DPO values of 1.191, 1.620, 2.164, and 5.725 ($n=5,20,50,100$). However, the $n=50$ HPS row lists BLEU/Reward as 0.230/0.307, sharply inconsistent with neighboring rows around 0.31/0.407 (HTML lines 704–713), suggesting a typesetting or transcription error. No trend is inferred from these two cells until the authors clarify them.

Beta sensitivity. Appendix Table 8 gives $\beta=0.1,0.25,0.5,0.75,1$. HPS RM_R-DPO values are 2.040, −0.828, −0.911, −0.925, −1.178; the value is positive only at 0.1. HPS exceeds BT at every beta, but “higher” does not mean “absolutely safe”: moving from 0.1 to 0.25 changes HPS RM_DPO from 2.723 to −0.146, a clear sign reversal. KL also falls from 11.767 to 6.972, indicating that the largest margin accompanies the greatest policy drift. The paper summarizes this as “all KL values outperform BT” (lines 730–731), without discussing this threshold.

gamma。 The paper provides only the −5→5 schedule and reports no ablations of fixed $\gamma$, schedule direction, or update frequency, nor tests of numerical stability for $gamma\to\infty$ . Thus, experiments do not establish that “sharper hardness is always better”; the gap between the theoretical limit and the best engineering setting remains unexplored.

Gaps in component attribution. Key controls are missing: “100 candidates + uniform single sampling,” “deterministic top-1 from the same pool,” “easy-to-hard schedule alone,” and “oracle human hardness.” The observed gains therefore cannot be fully decomposed into five components: multiple candidates, external RM ranking, single sampling, temperature curriculum, and the HPS loss.

Statistical significance

The paper reports a mean±variation only for training time: PL 168.4±1.9、BT 62.8±1.1、HPS 64.4±0.8 GPU-hour. Main-table BLEU, Reward, margins, and human evaluation are point estimates; random-seed counts, standard deviations, confidence intervals, bootstrap estimates, paired tests, and multiple-comparison corrections are not reported. Although human evaluation includes 20 raters and 30 prompts, inter-rater agreement, question-level variance, and effect intervals are also absent. One can therefore say HPS scores higher in this run, but cannot rigorously claim that +0.11 Likert is statistically significant.

Three decimal places in Qwen win rates do not imply high precision. Without total sample counts, paired repetition structure, and judge position-bias estimates, the difference between 63.875% and 58.712% cannot be interpreted at the displayed precision. Best practice would apply paired bootstrap over prompts and separately report judge seeds, swapped response positions, and single-sample versus best-of-5 results.

Actual contributions, reusable designs, and transfer potential

The actual contribution is not another preference loss, but decomposing “listwise information” intoan offline ranking distribution + online single-negative updates. This lets any trainer accepting only (prompt, chosen, rejected) exploit 100 candidates without paying PL’s full backpropagation cost. A second contribution formalizes the hardest-negative max-margin intuition in alignment theory; a third demonstrates orthogonal plug-in use across DPO/EXO/IPO/SPPO/NCA.

Reusable designs include replacing $q$ with a safety classifier, factuality verifier, or multi-objective Pareto distribution; caching sampling probabilities with a small RM; setting $gamma$by domain or user group; and adding newly failed responses to the candidate pool online. In code generation,$r_{est}$ can use unit tests; mathematical reasoning can use a process verifier; medical QA can use a multi-objective score combining guideline consistency and refusal quality. In any high-risk domain, however, the definition of a hard negative must be calibrated with external empirical evidence; a “high model score” cannot directly mean “most dangerous.”

Limitations stated by the paper

1. Evaluator budget (Section 7, HTML lines 337–340). The authors explicitly say budget constraints led them to estimate win rates with an open-source LLM; stronger GPT-4/Claude models might be more reliable. This limits the judge’s external validity.
2. Compute constraints (Implementation, HTML lines 313–316). Compute limits restrict experiments to LoRA rank 8; conclusions for full-parameter training, larger models, and other architectures are therefore unverified.
3. Not explicitly stated in Limitations, but directly bounded by the setup. Only HH-RLHF and PKU-SafeRLHF, one Llama3 reference family, and 2048-token contexts are used, leaving long contexts, multilingual use, and production red-teaming uncovered. The paper’s limitations section lists only one item; the additional two above are attributed to the main-text setup rather than presented as the authors’ own limitation statements.

Independent critique

1. [Construct validity] Because RM_DPO is derived directly from $log\pi_\theta/\pi_{ref}$ and HPS Theorem 3 and its training objective directly maximize that margin, part of the large margin increase reflects an objective–metric equivalence; it demonstrates successful optimization but cannot alone establish reduced real-world harmful generation.
2. [External validity] Because all 100 candidates per prompt are expanded by one DPO model and ranked by Skywork RM, the actual human information still comes mainly from the original binary pairs. “Hard human preference” is thus more accurately “hard reward-model preference,” which may fail to transfer across generators or cultural groups.
3. [Evaluation bias] Because each method samples 5 responses and reports the highest value for each metric (HTML line 313), whereas production usually returns one sample, Table 2 is a best-of-5 upper bound and cannot directly represent a single-sample deployment safety rate.
4. [Hyperparameter robustness] Because HPS RM_R-DPO is 2.040 at $\beta=0.1$ but becomes −0.828 at 0.25, and the largest margin corresponds to KL=11.767, the main conclusion depends on a narrow hyperparameter region. “Robust rejection” therefore needs comparison at fixed KL or fixed external quality.
5. [Theory–implementation consistency] Because the method definition specifies $gamma>1$but implementation starts at −5, negative temperature reverses the ranking and initially samples easy negatives. The hardest-distribution limit of Theorem 2 therefore does not cover the full training path, so the theory cannot explain gains during the first half of the curriculum.
6. [Attribution] Because uniform single sampling, deterministic top-1, fixed gamma, and oracle ranking controls are absent, single-sample efficiency is not separated from hard weighting. The 64.4 GPU-hour cost and margin gains therefore cannot all be attributed to “hard sampling.”
7. [Statistics] Because the main table and human evaluation lack variance/confidence intervals, human evaluation includes only 30 prompts, and DPO-HPS exceeds BT by just 0.11 Likert, the direction can be reported but the effect cannot be declared stable or significant.
8. [Data consistency] Because Appendix Table 7’s $n=50$ HPS BLEU/Reward of 0.230/0.307 is inconsistent with neighboring values of 0.31/0.407, at least two cells may contain transcription errors. Any “smooth scaling law” based on this point is therefore unreliable.

Related work: origins of the problem and technical lineage

                         Human/AI preference data
                                |
               +----------------+----------------+
               |                                 |
          Binary chosen/rejected            Multi-response rankings
               |                                 |
         DPO / KTO (2023–24)          PRO / LiPO (2024–25)
               |                                 |
               |                    More information, but costly full-list backprop
               |                                 |
               +---------------+-----------------+
                               |
                  HPS: rankings only construct q
                               |
              +----------------+----------------+
              |                                 |
      Single-negative updates: BT-level cost    γ→∞: maximin margin
              |                                 |
     Plug-in DPO/EXO/IPO/SPPO/NCA       Safety implications need external validation

Open challenges: actionable gaps for head-to-head experiments

Recent-work overview

1. Policy-labeled Preference Learning(ICML 2025) incorporates behavior-policy labels into a regret-based likelihood, addressing behavior-policy mismatch in offline preference data. It reminds us that HPS’s 100 candidates come from a specific generation policy and that generator information should not be discarded.
2. AMPO(ICML 2025) uses active subset selection to cover reward extremes and semantic clusters. A direct comparison with HPS should fix subset size and distinguish “sampling only the hardest” from “also preserving diversity.”
3. Energy-Based Preference Model / EPA(ICML 2025) notes that BT MLE may be non-unique and approximates energy models using strong negatives plus many free weak negatives. It challenges the scalar ranking underlying HPS from the perspective of preference-model identifiability.
4. Beyond Bradley-Terry Models / GPM(ICML 2025) represents cyclic preferences with vector embeddings while retaining O(K) query complexity. It is suitable for testing whether HPS incorrectly forces a linear ordering on intransitive preferences.
5. Right Now, Wrong Then(ICML 2025 PMLR volume contents) studies nonstationary DPO under preference drift. For HPS, cached $q$ can become invalid across populations or over time; timestamps and online re-estimation should be added.
6. PoisonBench(ICML 2025 PMLR volume contents) evaluates preference-data poisoning. HPS’s concentrated sampling of high-scoring hard negatives may amplify attacks; test whether a few poisoned responses dominate $q$。

Conclusion

HPS’s most valuable design takeaway is:Use multi-candidate information for sampling without sending every candidate through backpropagation. Under the same 8B/LoRA setup, it reduces PL fine-tuning time by 61.76% and enlarges implicit margins across multiple optimizers; its human rating of 3.93 also exceeds BT’s 3.82, indicating that it is more than a purely numerical exercise.

The key caution is equally clear: DPO-HPS HH Reward is 0.430, almost identical to BT’s 0.431; the largest numerical jump occurs in a margin directly shaped by the objective. Together with best-of-5, a narrow beta phase transition,$gamma$ definition–implementation inconsistency, and missing significance tests, HPS should be positioned asan efficient hard-negative alignment framework, rather than a solution with a completed real-world safety validation loop.

Index and production footnotes. The date 2026-09-21, LLM/NLP, HPS, ICML 2025, and a one-sentence conclusion have been appended to ml-report-index.md. The HTML uses single-file inline CSS/JS, system-ui + PingFang SC/Microsoft YaHei, a warm paper-like palette, light/dark theme switching, responsive wide tables, and keyboard-focusable SVG tooltips; its content matches the Markdown/plain text. The Warm Editorial structure validator passed with 0 warnings; this machine lacks /usr/bin/chromium，/opt/pw-browsers and an executable Chromium, so actual Playwright light/dark/mobile screenshot checks could not be completed. Static checks of responsiveness, themes, and dependencies were substituted; browser pixel-level acceptance testing cannot be claimed. All URLs were actually visited in this round; unavailable or missing items are explicitly marked.