MM-RLHF: Multimodal Preference Alignment and Its Evidence Boundaries

Core paper: Yi‑Fan Zhang et al. MM‑RLHF: The Next Step Forward in Multimodal LLM Alignment. ICML 2025 Main，PMLR 267:76625–76654。
Report date: 2026‑09‑15　Field: Vision and Multimodality　Evidence convention: Prioritize final ICML/PMLR; arXiv has only v1. Separate this report's recomputations from original values.

0. Verified links and version audit

All URLs were accessed on 2026-09-15. Cloudflare blocked OpenReview, so only the PMLR-listed destination is confirmed; review content was not read. Three CVPR OpenAccess entries returned parser errors, so corresponding arXiv/GitHub sources cross-checked facts. Not public/not found describes accessible entries that day.

Three inconsistent public counts

Final ICML Table 1 gives 29,997 queries, producing, according to the abstract, 120k comparison pairs. GitHub README Key Components instead says 20k instructions / 80k pairs, while the Hugging Face viewer showed approximately 16.3k rowstraining rows. These may reflect full, cleaned, and viewable releases, but no stage-wise manifests or hash mappings explain them. The difference is at least 29,997→20,000, down 9,997 or 33.33%, then roughly 16,300, about 45.66% below final-paper counts (recomputed). Public data does not automatically mean the 120k training pairs can be reconstructed exactly.

1. Candidate pool, deduplication, and random selection

First check ml-report-index.md, with recent records excluding AVUT (audio-video shortcuts), DEFAME (multimodal fact-checking), Ambient Creativity (generative memory), VIKI-R (multi-robot), and ThinkSLM (small-model reasoning evaluation). This round used ICML 2025 PMLR proceedings、EMNLP 2025 ACL Anthology, arXiv, OpenReview, Hugging Face Daily Papers, Papers with Code, and Semantic Scholar for title/version checks. The latter discovery/verification sources supplement official conference pages and PDFs as quantitative evidence.

The fixed pool contains 24 papers:

1. AGAV‑Rater: Adapting Large Multimodal Model for AI‑Generated Audio‑Visual Quality Assessment
2. CtrlSynth: Controllable Image Text Synthesis for Data‑Efficient Multimodal Learning
3. Sample‑level Early Feature Enhancement for Long‑Tail Visual Recognition
4. On Path to Multimodal Generalist: General‑Level and General‑Bench
5. Gradient Inversion of Multimodal Models
6. Imagine while Reasoning in Space: Multimodal Visualization‑of‑Thought
7. AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
8. OV‑MER: Towards Open‑Vocabulary Multimodal Emotion Recognition
9. Decomposition of Graphic Design with Multimodal Large Language Models
10. Graph4MM: Weaving Multimodal Learning with Structural Information
11. LMAct: A Benchmark for In‑Context Imitation Learning with Long Multimodal Demonstrations
12. SK‑VQA: Synthetic Knowledge Generation at Scale for Training Large Multimodal Models
13. A Multimodal Medical Code Tokenizer for Structured Clinical Prediction
14. MM‑RLHF: The Next Step Forward in Multimodal LLM Alignment
15. TRUST‑VLM: Thorough Red‑Teaming for Uncovering Safety Threats in Vision‑Language Models
16. Robust Multimodal Large Language Models Against Modality Conflict
17. MME‑CoT: Benchmarking Chain‑of‑Thought in Multimodal Models
18. Do Vision‑Language Models Really Understand Visual Language?
19. ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding
20. Layer‑wise Alignment for Multimodal Large Language Models
21. VisiPruner
22. Looking Beyond Text
23. Vision‑Free Retrieval
24. VLA‑Mark

The system command executed was shuf -i 1-24 -n 1returned 14, selecting MM-RLHF. No redraw: the 30-page final ICML paper, code/data/weights, and two benchmark entries were public, with sufficient tables/comparisons/appendices.

2. Motivation: a unified cross-task pipeline, not merely more preference pairs

Multimodal post-training has three breaks. Many open MLLMs stop at SFT, while preference optimization concentrates on object hallucination/dialogue without establishing joint safety, OCR, mathematics, and video gains. Scalar rewards say good/bad without explaining errors, while generic MLLM judges must combine perception, critique, and stable scoring. Finally, ordinary DPO assigns every pair the same $\beta$, treating first-versus-fourth and third-versus-fourth ranks as equally strong evidence.

Three sets of numbers motivate the paper. Starting from about ten millionmultimodal instructions, clustering and deduplicated sampling select 29,997 queries. Over two months, 50+ annotators supervised by eight multimodal experts produce more than 120k ranked comparison pairs (pp.1–4, Table 1). Evaluation spans ten dimensions and 27 benchmarks(abstract, p.1), beyond one hallucination score. LLaVA-OV-7B's conversation dimension reportedly improves 19.5%, while Unsafety falls from 40.2 to 13.9, an absolute −26.3 pp and relative reduction of 65.42%(Figure 1, Appendix Table 9; relative value recomputed).

Motivation is not conclusion: Figure 1 also shows at least two negative aggregate dimensions, Hal Rate around +0.45%(lower is better, hence slight degradation) and high-resolution/RealWorld around −1.59%. The precise question is whether unified preference training spreads strong gains across most dimensions without sacrificing high-resolution and fine-grained perception.

3. Argument, core assumptions, and trade-offs

The narrative has four steps.

1. Coverage: resample the 10M pool by task type and visual clusters into long-answer, short-answer, MCQ, safety, and video queries.
2. Human judgment: score three to five strong-model answers per query for Helpful, Faithful, and Ethical qualities, then provide overall rankings/reasons. Annotators add positives for low-quality ties and construct erroneous negatives for high-quality ties.
3. Turn explanations into rewards: train a language head to generate critiques, then train a scoring head on query, response, and critique using pairwise preferences.
4. Turn rewards into update strength: compute reward margins for every ordered response pair and use a bounded function to adjust DPO's $\beta$, training on all unequal-rank pairs.

Core assumptions and costs:

The data–reward–policy loop is both the strongest design and its main risk. The same data ecosystem trains rewards, supplies RewardBench samples, and adjusts DPO. Without external distributions, internal consistency can be mistaken for broad generalization.

4. Equation-level formalization

Given multimodal query $x$, preferred response $y_w$ , and dispreferred response $y_l$, standard reward modeling uses Bradley–Terry loss:

$$\mathcal L_{\text{reward}}(\theta)= \mathbb E[-\log \sigma(r_\theta(y_w\mid x)-r_\theta(y_l\mid x))].$$

MM-RLHF separates critique generation from conditional scoring. For human/expanded critique $c$, the language head minimizes:

$$\mathcal L_{\text{critique}}=-\mathbb E\sum_{t=1}^{\lvert c\rvert} \log \pi_\theta(c_t\mid c_{<t},x,y),$$

The score head uses ground-truth critiques through teacher forcing during training:

$$\mathcal L_{\text{score}}= \mathbb E[-\log\sigma(r_\theta(x,y_w,c_w)-r_\theta(x,y_l,c_l))], \qquad \mathcal L_{\text{total}}=\mathcal L_{\text{critique}}+\mathcal L_{\text{score}}.$$

At inference, first generate $\hat c\sim\pi_\theta(\cdot\mid x,y)$, then output $r_\theta(x,y,\hat c)$, changing the input distribution. Final Table 2's inference-with-GT-annotation raises ACC+ from 0.67 to 0.87, +0.20 absolute or +29.85% relative (recomputed), directly identifying critique generation rather than scoring as the current bottleneck.

Standard DPO is:

$$\mathcal L_{\text{DPO}}=-\mathbb E\log\sigma\left[ \beta\left( \log\frac{\pi_\theta(y_w\mid x)}{\pi_{ref}(y_w\mid x)}- \log\frac{\pi_\theta(y_l\mid x)}{\pi_{ref}(y_l\mid x)} \right)\right].$$

First calculate margin $\delta=r(y_w)-r(y_l)$, then set:

$$\beta(\delta)=\beta_{ori}\bigl[1+w(1-e^{-k\delta})\bigr], \quad \beta(\delta)\in[\beta_{ori},(1+w)\beta_{ori}].$$

Default $\beta_{ori}=0.1,w=0.5,k=0.5$(Appendix F, pp.28–29). Caution: although called loss weight/update strength, $\beta$ appears inside the sigmoid and also has a KL/temperature interpretation. It does not simply multiply sample loss; only at the local-gradient level does larger $\beta$ typically impose steeper pairwise-margin pressure.

Training pseudocode

Input: query x; three to five responses with human rankings; reference policy π_ref
1. Enumerate all unequal-rank (y_w, y_l); do not treat ties as preferences
2. Generate/read human critiques; GPT-4o only expands existing reasons
3. Jointly train critique and score heads
4. Obtain δ = r(y_w)-r(y_l) from Reward-7B
5. Compute β(δ), bounded by [β_ori, (1+w)β_ori]
6. Update π_θ with dynamic-β DPO; choose model-specific SFT-loss weights
7. Evaluate dialogue, perception, OCR, mathematics, hallucination, video, multiple images, high resolution, and safety
Output: aligned MLLM, critique-generating reward model, and two benchmarks

5. Data and engineering details

5.1 Dataset composition

Original image-task proportions are Long 12.17%, Short 83.68%, MCQ 4.14%; sampling changes to 4:5:1. CLIP ViT-B/32 encodes images, KNN forms 100 clusters, and each is randomly sampled. Video uses first-frame features only (pp.3–4). Responses come from Qwen2-VL-72B, LLaVA-OV-72B, GPT-4o, Claude-3.5-Sonnet, plus LLaVA-Video-72B for video.

5.2 Key engineering settings

6. Benchmarks, metrics, and reading averages

The 27 benchmarks are aggregated into Conversation, Hallucination Score/Rate, General, Chart, OCR, Math, Multi-Image, Video, RealWorld, Safety, and Unsafety dimensions. They include LLaVA-Wild/Wilder, WildVision, MME/MMBench/MMStar/SeedBench/VQAv2/AI2D, ChartQA/DocVQA/InfoVQA/OCRBench/TextVQA/WebSRC, MathVista/MathVerse, POPE/MMHal/Object-Hallucination, Video-MME/VideoChatGPT/VideoDC, LLaVA-Interleave/MMMU-Pro, and safety attacks (Appendix Tables 8–9, pp.24–27).

RewardBench ACC is pairwise winner accuracy; stricter ACC+ requires every pair for a query to be ordered correctly. Safety tables mix higher-is-better recognition/refusal and lower-is-better ASR/Unsafety; averaging without preserving direction misleads.

7. Main matrix I: eight preference-optimization algorithms on identical data

The following is final Table 4 (p.8), with every method trained on MM-RLHF. Bold marks column bests. Baseline MathVista is 60.10 both in the earlier appendix and Table 4. This supplies more than five benchmarks×six methods, but the paper providesnostandard deviations or significance tests.

Against the strongest non-MM-DPO baseline, MM-DPO gains OCRBench 69.30−66.50=+2.80 pp(+4.21% relative), DocVQA 86.11−85.88=+0.23 pp(+0.27%), MathVista 61.60−61.10=+0.50 pp(+0.82%), and MMHal 4.08−3.95=+0.13(+3.29%); LLaVA-Wild ties LLaVA-Critic. Three of the four absolute gaps are at most 0.5, so without variance they are table bests, not statistically significant wins.

8. Main matrix II: separating data and algorithm contributions

Final Table 5 (p.9) fixes DPO and changes preference data, a closer dataset-level control than Figure 1.

MM-RLHF wins four of five columns, but trails MPO-Data on MathVista by 1.80 pp(−2.84% relative). MPO-Data's mathematics focus shows broad coverage cannot replace dense domain data. A reusable strategy is a general preference base with targeted domain mixing, rather than one dataset expected to dominate all metrics.

9. Reward model: strong internal scores, substantial external closed-model gaps

9.1 Complete RewardBench matrix

Final Table 2 (p.7); each cell is ACC/ACC+:

From w/o enhanced to full, ACC+ rises 0.57→0.67, +0.10 absolute/+17.54% relative; from w/o Task 1, +0.17/+34.00%. Yet full Safety ACC/ACC+ is .66/.17, below GPT-4o's .72/.33. Overall superiority cannot be extended to every category.

9.2 Cross-model comparison

Final Table 3 (p.8) averages General/Hallucination/Reasoning. MM-Reward-7B scores 45.04/50.45/57.55，Avg 50.15; the strongest open comparator, InternVL2-26B, has 39.30/36.90/60.80, Avg 45.67, yielding +4.48/+9.81%. It also beats GPT-4o-mini 44.80 by 5.35/+11.94%, but trails Claude-3.5-Sonnet 53.57, GPT-4o 62.40, and Gemini-1.5-Pro 62.50. The gap to GPT-4o remains 12.25 points / 19.63%and to Gemini 12.35 / 19.76%(all recomputed here).

The text says comparable to or exceeding GPT-4o, but the same page's Table 3 average does not support that. It may refer to selected internal RewardBench columns or another setting. The precise reading is:the 7B reward model has the highest open-model average in the table, but does not match GPT-4o on this external matrix.

10. Safety gains, local regressions, and completeness boundaries

Appendix Table 9 (pp.26–27) gives LLaVA-OV-7B Safety average 75.8→85.4(+9.6 pp, +12.66% relative), Unsafety 40.2→13.9(−26.3 pp, −65.42% relative). Specifically, NSFW-Text 60.4→10.6 (−49.8 pp, −82.45%), NSFW-Image 80.1→24.2 (−55.9 pp, −69.79%), and Multimodal ASR 41.2→8.3 (−31.9 pp, −77.43%). These are the strongest large effects.

Safety is not monotonically better: Risk 82→76 (−6 pp), targeted adversarial 37→40 (+3 pp, worse if attack success). Some subsets are tiny: cross-model ASR and RtA each contain 18 examples (Appendix Table 6, pp.22–23), so one example equals 5.56 pp. Aggregate safety +60% summarizes direction, not every threat model.

Main benchmarks also regress: Appendix Table 8 (pp.24–25) gives LLaVA-OV-7B MME-RealWorld English 48.36→46.95 (−1.41 pp), Chinese 54.01→53.39 (−0.62 pp), RealWorldQA 66.41→65.75 (−0.66 pp), AI2D 81.41→81.22 (−0.19 pp), MathVista format 62.50→62.20 (−0.30 pp). Improvement on nearly all benchmarks is an aggregate trend, not a cell-by-cell fact.

11. Ablations, sensitivity, and transitions

11.1 Three layers of critique contribution

Direct pairwise LLaVA-Critic ACC+ is only 0.15; adding human-critique training without enhancement yields 0.57, suggesting explain-then-score is a major step. GPT-4o expansion raises 0.57→0.67, but added content versus evaluator-style alignment cannot be separated. GT-critique inference reaches 0.87, exposing a 0.20 generation gap. These are not fully architecture/objective-matched single-variable changes, so differences cannot be summed as independent contributions.

11.2 Algorithm components

MM-DPO beats ordinary DPO in all five columns, but there is no full-data orthogonal comparison of all pairs without dynamic $\beta$versus dynamic $\beta$with only hardest pairs. Table 4 compares beta-DPO, SimPO, LLaVA-Critic, MIA-DPO, and MPO, establishing recipe competitiveness without fully separating all-pairs and Dynamic Reward Scaling contributions.

11.3 A small-model self-improvement transition

The paper samples up to 8 responses from the current policy, then uses the RM to select preference pairs (§5.3 and appendix). Small models often produce identical wrong MCQ answers; if the candidate support lacks a correct answer, no RM can create one. Three-stage LLaVA-Critic improves LLaVA-Wild/MMHal but reaches only OCRBench 64.20 and MathVista 58.74, below MM-DPO by 5.10 and 2.86 (Table 4, p.8). This is an on-policy support-set transition, not simply ranker failure: below a generation-quality threshold, self-training chooses among degrees of wrongness.

11.4 $k,w$ Sensitivity

Appendix Figure 12 (p.29) gives curves only; the conclusion is that $k,w$ should be neither too large nor too small, with both defaulting to 0.5;exact cell-readable scores, standard deviations, confidence intervals, and transition thresholds are unreported. Dynamic ranges need constraints, but universal optimality of 0.5 or a precise critical point is not established.

12. Statistical significance and reproducibility

Main results omit seed counts, standard deviations, CIs, significance tests, and multiple-comparison correction. Table 4 gaps of +0.13, +0.23, and +0.50 may lie within ordinary training noise. Unsafety −26.3 pp and NSFW-Text −49.8 pp are large effects worth reproducing even without error bars.

Public code/model/data entries are a strength, but four gaps remain: inconsistent counts of 29,997/20k/16.3k; hardware reported as 32×H800 without duration; GPT-4o expansions and closed-source answer generators may change with service versions; and OpenReview reviews were inaccessible that day. The framework is reusable, but all tables are not one-click reproducible.

13. Related-work chain: solved problems, remaining gaps, and MM-RLHF's response

13.1 DPO: converting RLHF to classification while treating pair evidence uniformly

Problem addressed: DPO derives a pairwise logistic objective from the optimal KL-constrained RLHF policy, removing explicit reward modeling and online RL (NeurIPS 2023, Eq.7).Remaining gap: The standard form fixes $\beta$, without distinguishing multimodal-pair confidence or using full rankings of three to five answers per query.MM-RLHF's response: enumerate all unequal-rank pairs and derive bounded $\beta(\delta)$。Transferred design: from Reward-7B margins, retaining reference-relative log ratios while injecting instance-level evidence quality into temperature/margins.

13.2 LLaVA-RLHF / Fact-RLHF: first showing multimodal RLHF can reduce hallucination

Problem addressed: Fact-enhanced rewards reduce visual hallucination. The paper reports LLaVA-Bench reaching 94% of text GPT-4 versus the prior 87%, and 60% improvement on MMHal (arXiv 2309.14525, abstract/main results).Remaining gap: The focus is factuality/hallucination; data size and coverage cannot establish joint OCR, mathematics, video, and safety gains.MM-RLHF's response: Generates 120k pairs from 29,997 queries across five types and evaluates ten dimensions/27 benchmarks.Transferred design: Factuality rewards expand into Helpful/Faithful/Ethical human rationales.

13.3 VLFeedback: scaling preferences through AI annotation

Problem addressed: VLFeedback provides AI feedback for 82k multimodal instructions, reporting base-model perception/cognition improvements of 6.9%/9.5% (final EMNLP 2024 abstract/main table).Remaining gap: AI annotators and judges may share biases; expert fine-grained human reasons are missing.MM-RLHF's response: More than 50 human annotators first provide scores, ranks, and reasons; GPT-4o is restricted to expansion.Transferred design: The large multisource answer pool remains, but preferences become human-anchored and AI-expanded rather than AI-defined.

13.4 LLaVA-Critic: general multimodal judging and iterative DPO

Problem addressed: Trains LLaVA for pointwise/pairwise evaluation of multimodal answers, supporting repeated sampling, filtering, and DPO.Remaining gap: Critics can overfit; if small models never sample correct answers, iterative selection cannot solve the problem.MM-RLHF's response: Builds broader human benchmarks and critique rewards; offline human rankings avoid complete dependence on on-policy candidates.Transferred design: Retains critique as intermediate reasoning and places it directly in the joint training objective.

13.5 RLAIF-V: low-cost open AI feedback

Problem addressed: Constructs preferences from open-source AI feedback, reporting 80.7% lower object hallucination and 33.7% lower overall hallucination for a 7B model (arXiv 2405.17220, abstract).Remaining gap: Still hallucination-focused, with potential self-reinforcement of feedback errors.MM-RLHF's response: Anchors preferences in human rankings and covers safety, video, general tasks, OCR, and mathematics.Transferred design: Automated feedback can expand and scale data without replacing high-value human judgments.

Nonlinear technical evolution

                         ┌─ Fact-RLHF / RLAIF-V ── hallucination and factuality
SFT-only MLLM ──────────┤
                         ├─ VLFeedback ───────── large-scale AI preferences
                         │
Text RLHF ─ DPO ─────────┼─ MIA-DPO / MPO ─────── multiple images, mathematics, mixed objectives
                         │
                         └─ LLaVA-Critic ─────── critic → sampling → iterative DPO
                                      │
                                      ├── inadequate small-model candidate support (negative feedback)
                                      │
Fine-grained human ranks ─ critique learning ─┴── MM-RLHF
          │                                      ├─ all ranked pairs
          └─ constrained GPT-4o expansion ────────├─ reward-margin β(δ)
                                                 └─ ten dimensions / 27 benchmarks

Three shifts drive this lineage: from answering to fewer hallucinations, from single-task preferences to general evaluators, and from uniform scalar rewards to explanation/confidence-aware updates. MM-RLHF allocates human effort to defining preferences and models to expansion/scaling. Risk shifts from absent human supervision to circular dependence on closed-source expansion and internal benchmarks.

Cross-paper comparison

14. Genuine contributions, reusable design, and transfer potential

Three contributions stand out. Five query types and ranking reasons across 29,997 queries expand multimodal preferences beyond caption–image matching. GT-critique controls quantify the generation bottleneck, more useful than merely claiming interpretability. Tables 4/5 fix data and algorithm separately, beginning to distinguish good data from good optimization.

The most reusable design is not a particular exponential function, but a three-stage protocol:humans define error dimensions → models generate structured critiques → calibrated margins control update strength. Medical imaging could use evidence localization, clinical correctness, and risk levels; video agents should replace first-frame clustering with temporal embeddings; edge models should combine reward confidence with latency/energy constraints in pair weighting.

15. Author-stated limitations

1. Insufficient high-resolution data. The authors attribute limited RealWorld/high-resolution gains to scarce ultra-high-resolution examples and sampling optimized for image similarity rather than resolution (§5.1, pp.7–8).
2. Fine-grained annotations are underused. The conclusion notes current algorithms underuse dimension-specific scores and ranking rationales, motivating more advanced optimization (§6, p.9).
3. Cost of scaling human annotation. The authors acknowledge expensive labeling and propose semiautomatic expansion (§2.3/§6, pp.4,9).

16. Independent critique: evidence → inference

1. [Dataset versions] Because the final paper gives 29,997 queries/120k pairs, README 20k/80k, and HF viewer about 16.3k rows without stage manifests, one cannot establish that released data match training data or recompute the effective all-unequal-rank-pair count.
2. [Evaluation loop] Because RewardBench samples ten examples per MM-RLHF category, totaling 170 pairs, while related data train Reward-7B, Table 2's 0.85/0.67 establishes within-ecosystem adaptation more directly than cross-distribution reward generalization.
3. [Training–inference mismatch] Because scoring trains on GT critiques but deploys on generated ones, and GT inference raises ACC+ 0.67→0.87, an observable 0.20 exposure gap remains. Stability depends on critique generation, not only the reward head.
4. [Teacher/judge dependence] Because GPT-4o expands training rationales and judges several open-ended benchmarks, gains may mix genuine human preference with GPT-4o-like critique style. A teacher×judge 2×2 experiment is needed to separate them.
5. [Video representativeness] Because clustering 4,235 videos uses only first frames, similar openings with different later events may share clusters. Video coverage does not establish temporal diversity.
6. [Safety averages conceal regressions] Because Unsafety drops while Risk falls 82→76 and targeted attacks rise 37→40, with two cross-model subsets of only 18 examples each, aggregate gains cannot imply improvement for every attack family or comprehensive robustness.
7. [Optimization attribution] Because MM-DPO combines all pairs, reward-margin-dependent $\beta$, model-specific SFT weights, and grid search without a complete orthogonal ablation, Table 4 gains cannot be uniquely credited to Dynamic Reward Scaling.
8. [Statistical evidence] Because main tables lack seeds, standard deviations, CIs, or tests, and DocVQA/MathVista/MMHal margins are only +0.23/+0.50/+0.13, they show single reported table leads, not stable SOTA.

17. Open challenges as executable head-to-head comparisons

Theory

• Calibrated versus raw margins: With identical queries/pairs/seeds, compare raw $\delta$, task-wise temperature-calibrated $\tilde\delta$, and binary confidence-interval lower bounds. Bucket by RM calibration error and report win rate/KL. OOD-only gains from calibration would support cross-task scale mismatch.
• Dynamic $\beta$ versus explicit sample weights: compare $\sigma(\beta_i z_i)$ with $a_i\log\sigma(\beta z_i)$while matching average gradient norms and KL budgets, testing whether the claimed loss weight is equivalent to importance weighting.
• Effective sample size of correlated pairs: Fix 120k pairs; compare one pair per query from more queries versus all pairs from fewer queries. Use query-cluster bootstrap CIs to test whether pair expansion merely repeats information.

Engineering applications

• Full reproducibility audit: Train the available final-paper 29,997, README 20k, and HF 16.3k versions under equal 32×H800-equivalent token/GPU-hour budgets. Different curves would make missing released data a reproduction variable, not just documentation.
• Compute-matched data×algorithm 2×2: Compare MM-RLHF/best external data × DPO/MM-DPO with equal tokens, search trials, and at least five seeds. Report means, 95% CIs, TTFT, and GPU-hours to separate data quality from tuning effort.
• Temporal video sampling: Keep 4,235 queries fixed and compare first-frame CLIP, uniform eight-frame pooling, and a video encoder. Report temporal reasoning separately from static VQA to test first-frame clustering's blind spot.
• Safety tail risk: Use at least 1,000 examples per attack family for base/DPO/MM-DPO. Report ASR, false refusals, calibration, and worst-group CVaR so Safety/Unsafety means cannot hide Risk regressions.

Emerging directions

• Verifiable critiques: Require image regions/video timestamps in critiques. Compare plain-text, grounded, and no-critique variants; remove cited evidence and measure reward changes to assess faithfulness rather than fluency.
• Repair small-model candidate support: At the same eight-sample budget, compare best-of-eight, retrieval-augmented candidates, one injected strong-teacher candidate, and offline human pairs. Stratify by whether a correct answer exists in the set to locate negative bootstrapping's bottleneck.
• Multiple-reward uncertainty: Compare one Reward-7B with five heterogeneous RMs; use disagreement to control $\beta$and test whether RealWorld/Risk regressions shrink without sacrificing dialogue gains.

18. Recent-work radar

1. MIA‑DPO（ICLR 2025）: specializes in multi-image augmented preference optimization, reporting about +3.0%/+4.3% for LLaVA-v1.5/InternLM-XC2.5. It tests the transferability of MM-RLHF gains without dedicated multi-image data.
2. OPA‑DPO（CVPR 2025 Oral）: on-policy correction addresses candidate/policy mismatch, further reducing AMBER/Object-Hal hallucination with only 4.8k examples. It directly targets limited self-sampling support but remains hallucination-focused.
3. RLAIF‑V（CVPR 2025 Highlight）: low-cost open AI feedback with large hallucination reductions is the key cost comparator for human-intensive MM-RLHF. The missing comparison matches GPU-hours and human hours.
4. SPA‑VL（CVPR 2025）: dedicated vision-language safety preferences could supplement Risk/targeted-attack weaknesses rather than be replaced by a general pool.
5. LLaVA‑Critic（CVPR 2025）: connects general multimodal critics with iterative DPO; MM-RLHF shows even strong critics remain constrained by candidate support.
6. VLA‑Mark（EMNLP 2025）: extends alignment/ownership into vision-language-action watermarking, showing trustworthy post-training expanding from safe answers to action-model governance.
7. Looking Beyond Text（EMNLP 2025）: studies whether multimodal evidence actually drives behavior, useful for auditing visual dependence of rewards versus critique-template matching.

19. Final assessment

MM-RLHF deserves attention for integrating a usually fragmented problem:which data to annotate, how reasons become rewards, how reward gaps control policy updates, and which dimensions define acceptance. Tables 4/5 support both data and algorithm benefits, with striking safety effects; GT-critique ablation identifies the next bottleneck.

It does not establish comprehensive multimodal alignment. Human-anchored, broadly covered offline preferences let MM-DPO improve dialogue, safety, OCR, and other dimensions together, while high resolution, some risk families, and specialized mathematics may regress. Reuse should prioritize fine-grained human protocols + critique/score separation + margin calibrationand immediately add external RewardBench evaluation, version manifests, compute-matched 2×2 controls, and multiple seeds.

Evidence footnote. Quantitative conclusions come from the read final ICML/PMLR PDF or individual related papers. Unreadable sensitivity-curve values were not filled in.ml-report-index.md This entry was appended locally; if persistent-index synchronization fails, reappend from this report. The interactive chart redraws only final Table 3 values. Warm Editorial structural validation passed. Playwright is installed, but neither /usr/bin/chromium、/opt/pw-browsers nor its cache provides a browser; actual launch failed, so light/dark/mobile screenshot checks were not completed or falsely claimed. Responsive tables, theme tokens, native MathML, and static accessibility checks remain.