MMedPO: Clinically Aware Multimodal Preference Optimization

0. Link verification table

All links below were accessed when this report was generated. “Not found” means unconfirmed in the paper, official project, or verifiable search results, not that the resource definitively does not exist.

1. Search, deduplication, and random-selection record

Today, date +%u returned 2, rotating the topic to “Vision and Multimodality.” Before searching, the following was read: ml-report-index.md; the past 14 days cover EPG, EARL-BO, DuoAttention, sparse-autoencoder theory, DPLM2.1, HEX, and other topics, with neither medical vision-language preference optimization nor the specific papers below, so no duplication conflict exists. Candidates prioritize official ICML 2025, NeurIPS 2025, and ICLR 2026 work from the past 12 months; a few rely on official OpenReview acceptance pages.

The random-selection command was shuf -i 1-26 -n 1 returned 12. MMedPO's official paper, full HTML text, code, main results, ablations, and engineering configurations were accessible, so the number of redraws was 0. The candidate pool provides auditable boundaries for random selection, not a claim that every candidate has equally mature supporting materials.

2. In-depth analysis of the core paper

2.1 Motivation: the issue is not “can it speak medicine?” but “does it actually use the image?”

Med-LVLMs such as LLaVA-Med connect visual encoders to language models and generate fluent medical answers, but fluency is not image fidelity. MMedPO identifies the specific failure of modality misalignment: training and decoding more readily rely on medical co-occurrence knowledge already stored in the language model than lesion evidence in the current X-ray or clinical image. A medically plausible description may therefore be false for this image. Ordinary textual DPO compares two answers for the same image; if the rejected answer is too absurd, the model can distinguish it using linguistic common sense without improving image understanding.

The paper quantifies this gap across four test sets. Base LLaVA-Med-1.5-7B scores on SLAKE open/closed questions are 44.26 / 61.30, and on VQA-RAD, 29.24 / 63.97; for IU-Xray report generation, average BLEU / ROUGE-L / METEOR are only 14.56 / 10.31 / 10.95, with corresponding MIMIC-CXR values of 10.25 / 9.38 / 7.71(core paper Table 1, LLaVA-Med row). These are not direct hallucination rates, but show substantial room for improvement in both question answering and report generation.

The novelty is that POVID already combined manually injected text hallucinations and whole-image perturbations in visual DPO, SIMA improved self-critique with visual metrics, and STLLaVA-Med introduced GPT-4o-supervised self-training to medicine. MMedPO does not simply stack these ideas; it makes a medical-specific judgment: not all errors or pixels are equally clinically relevant. It perturbs only lesion regions and weights preference samples using medical-model ensembles or visual tools, aiming optimization at errors that affect clinical judgment.

Why solve this? Medical image-answering errors have asymmetric costs: an inelegant background description and an invented lesion are different risks. Annotating every preference pair is also expensive and consumes expert time. Tool-generated hard negatives combined with inexpensive LoRA fine-tuning could turn domain alignment from costly annotation into a reusable data–optimization workflow. However, this paper does not validate real clinical-decision safety; its value is a research framework, not authorization to deploy.

2.2 Narrative: three turns from hard negatives to clinical weighting

The first turn treats hallucinated answers as preference data. The authors sample responses from the target model and ask GPT-4o to select the most hallucinated examples. If samples do not yield sufficiently hard negatives, GPT-4o generates medically plausible but factually incorrect answers. Chosen versus rejected then differs in clinical facts rather than grammatical quality.

The second turn acknowledges that textual negatives may still let the model bypass images. MedKLIP produces lesion heatmaps using the prompt “disease,” and diffusion noise is injected only into lesion regions. For the same answer, replacing the original image with a locally corrupted image forces DPO to exploit differences in visual evidence. This sacrifices coverage of unknown lesions and tool misses in exchange for semantically concentrated perturbations.

The third turn weights preference pairs. Three Med-LLMs score text negatives through multiround debate, while MedKLIP confidence scores visual negatives. Standardized, clipped scores become loss weights. The implicit assumption is that tool consensus/confidence increases monotonically with clinical importance. Intuitive though it is, physician ratings do not externally calibrate this assumption, making it a central limitation below.

The overall workflow is:

                                        ┌─ Repeated target Med-LVLM samples ─ GPT-4o selects/creates hard hallucinations ─ Med-LLM ensemble scoring ┐
Original (image, question, correct answer) ┤                                                                                      ├─ Weighted preference pairs
                    └─ MedKLIP lesion heatmap ─ Diffusion noise only in lesion regions ─ Tool-confidence scoring ───────────────┘
                                               ↓
                                      Clinical-aware M-DPO
                                               ↓
                                    LoRA updates LLaVA-Med-1.5-7B

2.3 Dissecting the methodology and mechanisms

Let the original image be (x_v), lesion heatmap/mask (h), cumulative noise coefficient at diffusion step (k) be (\bar\xi_k), and Gaussian noise (\epsilon). The visual rejected sample is:

[ x_v^*=\sqrt{\bar\xi_k}(x_v\odot h)+\sqrt{1-\bar\xi_k}(\epsilon\odot h)+x_v\odot(1-h). ]

The key is not standard diffusion noise, but the final term preserving non-lesion regions intact: it changes global image degradation into local distortion of clinical evidence. Textual negatives generate (y_l) under the original image, with correct answer (y_w). Tools supply a raw relevance score (s), standardized with (\mu=1,\sigma^2=0.1) and clipped:

[ s' = \operatorname{clip}\left(\frac{s-\mu}{\sigma},0.75,1.25\right). ]

The final loss expands DPO's implicit reward difference, allowing the rejected condition to involve either text or perturbed visual input:

[ \mathcal L_{\text{MMedPO}}=-\mathbb E\left[s'\log\sigma\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)}-\beta\log\frac{\pi_\theta(y_l|x^)}{\pi_0(y_l|x^)}\right)\right]. ]

Here (\beta) denotes DPO temperature, avoiding conflict with the paper's use of (\alpha) for the clipping lower bound. Intuitively, when tools judge a preference pair more clinically relevant, (s') increases and amplifies its gradient. Clipping limits maximum/minimum weights to 1.25/0.75, a maximum ratio of 1.667, preventing tool noise from completely dominating training.

Pseudocode summary:

for each clinical example (image, prompt, preferred_answer):
    candidates = target_med_vlm.sample(image, prompt)
    text_negative = GPT4o.select_or_generate_hard_hallucination(candidates)
    text_score = debate([Med42-7B, Med42-70B, BioMistral-7B], max_rounds=5)

    lesion_map, visual_score = MedKLIP(image, prompt="disease")
    corrupted_image = diffuse_only_inside(image, lesion_map)

    add weighted pair (image, preferred_answer, text_negative, normalize(text_score))
    add weighted pair (corrupted_image, preferred_answer, rejected_condition, normalize(visual_score))

optimize weighted_DPO_with_LoRA(epochs=3, lr=1e-7, batch_size=4)

Preference training-pair counts: IU-Xray visual 2,069 + text 2,069 = 4,138; MIMIC-CXR 800 + 800 = 1,600; SLAKE 4,919 + 4,919 = 9,838; VQA-RAD 1,797 + 1,797 = 3,594(core paper experimental setup / data-statistics table). Total: 19,170 pairs, split exactly equally between visual and textual pairs. This is a clear engineering choice, but does not establish that 1:1 is optimal.

2.4 Benchmarks, datasets, and metrics

2.5 Main results: complete 10-metric matrix

All values below are transcribed from the core paper's Table 1 (non-SFT group). Bold indicates the best value per column. The final column is this report's simple mean of 10 percentage-scale metrics with different meanings, used only as an overall check; it is not an official paper metric.

The most rigorous head-to-head comparison is not MMedPO versus original DPO, but against Table 1's strongest non-MMedPO competitor in each column:

The strongest quantitative evidence in the paper is not an inflated percentage, but that MMedPO exceeds the actual column-wise strongest competitor in all 10/10 columns, with absolute advantages of 1.30–3.80 pp. IU-Xray ROUGE-L's +3.80pp is the largest absolute gain; MIMIC-CXR METEOR's +27.93% is the largest relative gain. The smallest advantage is only 1.30pp, however, and without error bars it cannot automatically be considered statistically significant.

The SFT group also warrants careful comparison: SFT+MMedPO leads on both SLAKE columns, VQA-RAD Closed, IU BLEU, IU ROUGE-L, IU METEOR, and all three MIMIC columns in Table 1, while VQA-RAD Open is 34.03. Is that below SFT+STLLaVA-Med's 33.72 ? No—it remains 0.31 higher, so it also leads every column. Some advantages are only fractions of a point, however, and still require repeated runs.

2.6 Ablations: how much each component contributes

Table 2 compares the textual stage (Stage 1), visual stage (Stage 2), and clinical relevance scoring (CRS) in matched pairs; values are aggregated dataset scores.

CRS has a strong effect on IU-Xray but almost none on MIMIC-CXR. “Clinical weighting is universally effective” is therefore too broad; the data supports effectiveness on some report-generation distributions, with cross-institution stability unproven.

Multiagent scoring (Table 3) versus single-agent scoring: SLAKE 56.09→57.53 (+1.44), VQA-RAD 48.67→51.14 (+2.47), IU-Xray 15.67→15.86 (+0.19), MIMIC-CXR 6.58→6.86 (+0.28). Local lesion perturbations versus global perturbations (Table 4): 58.88→59.88, 46.91→46.98, 24.88→25.00, 6.80→7.24, or +1.00, +0.07, +0.12, and +0.44pp. The two smallest effects are tiny; without variance, whether they exceed random fluctuations is unknown.

Finer report-generation ablations (Table 8) are nonmonotonic: IU-Xray Stage 2-only METEOR is 34.26, while full Stage 1+2 with multiple agents gives 34.16, lower by 0.10; MIMIC-CXR instead rises from Stage 2-only's 8.81 to approximately 10.05 with the full configuration. Textual and visual negatives are thus not always complementary, showing slight interference on at least one metric.

2.7 Hyperparameter sensitivity, phase transitions, and statistical significance

The paper reports a fixed training configuration (LoRA, 1e-7, batch 4, 3 epochs, 4×A6000, 2–3 hours), but no systematic sensitivity curves for learning rate, LoRA rank, diffusion steps, lesion thresholds, textual/visual negative ratios, CRS clipping intervals, or DPO temperature. An experimentally established phase transition therefore cannot be claimed.

There is a design-imposed hard threshold: CRS weights are clipped to [0.75,1.25], giving at most a 1.667× dynamic range. This prevents extreme judge scores from dominating gradients, but there is no sweep showing whether performance plateaus near 1.25. Local versus global perturbations improve the four datasets by only 0.07–1.00pp, insufficient to infer a jump beyond some lesion-localization threshold.

No multiseed means, standard deviations, confidence intervals, or significance tests are reported. MIMIC-CXR has only 200 test samples, and some SLAKE/VQA-RAD gains are just 1–2pp. Without sample-level bootstrap or at least 3 training runs, stable gains cannot be separated from sampling/decoding fluctuations. This report preserves decimal values without treating decimal places as statistical precision.

2.8 Genuine contributions, reusable designs, and transfer potential

There are three contributions. First, reframing: multimodal alignment cannot rely solely on response-text preferences because language priors can produce “correct” answers; negatives should cover both responses and evidence modalities. Second, domain structure: perturbations should follow task-relevant regions rather than arbitrary pixels—lesion maps in medicine, target regions in remote sensing, or defect masks in industrial inspection. Third, preference pairs are not equally valuable; bounded tool-confidence weights form an inexpensive, stable soft curriculum.

Reusable modules include hard-text-negative generators, diffusion perturbations within ROIs, bounded mapping of tool consensus to [0.75,1.25], a unified DPO interface for two-channel preference data, and the local-versus-global perturbation ablation template. For general VLMs, segmentation/grounding models could replace MedKLIP and task-expert models could replace Med-LLM judges. Weights must be recalibrated in the target domain; “tool confidence = clinical importance” cannot simply be transferred.

The most valuable design lesson is to make counterfactuals destroy only decision-relevant evidence. This is closer to causal intervention than generic noise: if an answer remains unchanged after destroying its evidence, the model may not have used that evidence. The next step is a testable counterfactual-consistency metric, rather than only a training trick.

2.9 Limitations: author statements and independent critique

Limitations stated by the paper

The paper has no separate Limitations/Discussion section and does not explicitly list 2–3 author-acknowledged limitations. That omission should be recorded honestly. Directly observable scope boundaries include using only LLaVA-Med-1.5-7B, only four datasets, and dependence on GPT-4o / three Med-LLMs / MedKLIP. These are constraints inferred here from the setup, not presented as author statements.

Independent critique (8 points)

1. Argument gap | Imprecise “best baseline” framing. POVID/FiSAO substantially exceed original DPO in most Table 1 columns, yet prominently stated relative gains mainly use DPO. Those percentages are therefore not head-to-head comparisons with the strongest competitors. Recalculation still yields wins in every column, but absolute gaps are 1.30–3.80pp; these should be the main conclusion.

2. Missing statistical significance. The paper provides no standard deviations, confidence intervals, or multiseed results, while local perturbations improve VQA-RAD/IU-Xray by just 0.07/0.12pp (Table 4). Stable gains on these two datasets cannot be established.

3. Metric validity. Report generation reports only BLEU, ROUGE-L, and METEOR, without CheXbert, RadGraph, or physician fact checks. Lexical metrics allow incorrect clinical facts expressed with similar words, so the experiments do not directly establish reduced clinical hallucination.

4. Experimental design | Narrow external generalization. Training and testing concentrate on IU-Xray, MIMIC-CXR, SLAKE, and VQA-RAD, with as few as 200 test samples. Results cannot be extrapolated to CT, MRI, pathology slides, fundus images, or cross-hospital device distributions.

5. Judge feedback-loop risk. GPT-4o selects/generates text negatives and a Med-LLM ensemble scores relevance without physician-blinded calibration. Optimization may learn shared machine-judge preferences rather than clinical preferences; up to 5 debate rounds may amplify shared-corpus biases.

6. Tool dependence and missed lesions. Visual negatives depend entirely on MedKLIP heatmaps from the single prompt “disease.” If the tool misses a rare lesion, training never perturbs the real evidence. Table 4's localization gains of only 0.07–1.00pp across four datasets are insufficient to establish precise localization as the main causal factor.

7. Modules are not strictly complementary. Table 8 reports IU-Xray Stage 2-only METEOR of 34.26, exceeding the full multiagent configuration's 34.16. Textual and visual negatives therefore interfere by -0.10 on at least this metric; no mixing-ratio sweep explains the conflict.

8. Incomplete reproducibility/cost reporting. Only total training time of 2–3 hours and 4×A6000 are given, without GPT-4o call counts, token costs, sampling temperature, diffusion steps, or LoRA rank. “Low cost” is supportable only for fine-tuning, not end-to-end data construction. Similar α symbols for different quantities further increase implementation ambiguity.

3. Related-work retrospective: the chain of problems and technical origins

3.1 LLaVA-Med — 2023, NeurIPS Datasets & Benchmarks

What it solved. LLaVA-Med uses PubMed Central image-text pairs and GPT-4-generated visual instructions, with a curriculum of medical-vocabulary alignment followed by open dialogue, to adapt general LLaVA to biomedicine. It reports training in under 15 hours on 8×A100 (LLaVA-Med arXiv abstract). It addresses the basic adaptation gap of general VLMs lacking medical image-text semantics.

Remaining gap. Instruction tuning optimizes resemblance to reference answers without directly requiring the current image to be read; a strong language model can answer from disease co-occurrences. Fluency can conceal evidence mismatch in open-ended QA.

How MMedPO responds. MMedPO directly uses LLaVA-Med-1.5-7B as its policy, moving from imparting domain knowledge to evidence-faithful alignment. It constructs counterfactual negatives during post-training rather than rebuilding the vision-language backbone.

Inherited design. It inherits the medical visual-encoder–projector–LLM architecture, four common datasets, and LoRA ecosystem; innovation occurs in preference data and loss.

3.2 DPO — 2023, NeurIPS

What it solved. DPO uses a closed-form parameterization of the optimal RLHF policy to turn “reward-model training + PPO” into a binary-classification-style objective over preference pairs, avoiding online sampling and complex RL tuning.

Remaining gap. Original DPO treats conditioning as a fixed textual prompt and assumes chosen/rejected examples faithfully encode human preferences. For VLMs, this does not guarantee preferences arise from images or distinguish clinically important from unimportant pairs.

How MMedPO responds. MMedPO retains DPO's reference-policy log-ratio structure, but extends rejected conditions to lesion-perturbed images and multiplies losses by clinical relevance scores.

Inherited design. It inherits implicit reward differences and reference-model KL constraints, evolving from equally weighted pairs with fixed conditions to intervenable conditions and bounded pair weights.

3.3 POVID — 2024, arXiv

What it solved. POVID explicitly frames VLM hallucination as modality misalignment: one branch uses GPT-4V to inject plausible hallucinations into correct answers; another distorts images to trigger inherent hallucinations; DPO jointly trains on both.

Remaining gap. Global or generic visual distortion can damage task-irrelevant pixels, while text negatives lack medical-risk ranking. Lesions may occupy tiny regions, so generic perturbations conflate poor image quality with disappearance of clinical evidence.

How MMedPO responds. It medicalizes POVID's two branches: GPT-4o supplies difficult clinical hallucinations, MedKLIP perturbs only lesions, and relevance weights are added to both sample types.

Inherited design. The dual-negative structure of externally injected and internally triggered hallucinations is inherited intact. Visual perturbations evolve from global degradation to ROI-local diffusion, and samples from equal to tool-based weights.

3.4 SIMA — 2025, NAACL Findings

What it solved. SIMA improves itself using model-generated candidates, self-critique, and visual metrics, directly targeting VLM language bias. It evaluates on 14 hallucination and comprehensive benchmarks (arXiv abstract).

Remaining gap. Generic visual metrics need not reflect medical importance, and self-critique can remain dominated by shared language biases. “Looks more consistent with the image” does not guarantee lesion-level correctness.

How MMedPO responds. It replaces generic visual critique with medical-specialist judge ensembles and MedKLIP, binding self-evaluation to clinical lesions and medical knowledge.

Inherited design. The generation–evaluation–preference-optimization loop is inherited; generic/single-model self-critique becomes heterogeneous medical tools and multimodel debate.

3.5 STLLaVA-Med — 2024, EMNLP main conference

What it solved. STLLaVA-Med automatically generates medical visual-instruction data with the policy model and supervises DPO using GPT-4o. Its abstract claims competitive zero-shot performance on three VQA benchmarks using only 9% of the medical data.

Remaining gap. It mainly addresses data scarcity, with preference supervision centered on answer/instruction quality. It does not explicitly construct controls with destroyed image evidence or continuous clinical-relevance weights.

How MMedPO responds. MMedPO retains GPT-4o supervision for text negatives and adds lesion-local visual negatives and CRS. It moves from learning medical answers with less data to more explicitly learning image dependence.

Inherited design. Three elements carry over: GPT-4o as an expensive expert, self-sampling by a smaller policy, and DPO LoRA fine-tuning. MMedPO adds multiple judges and visual tools, turning single-expert supervision into a composite system.

4. Technical evolution

2023 LLaVA-Med [medical visual-instruction tuning; domain knowledge/interface]
      │
      ├──────────── 2023 DPO [preference optimization without an explicit reward model]
      │                         │
      ↓                         ↓
2024 STLLaVA-Med [GPT-4o supervision, self-training, data efficiency] ──┐
      │                                                              │
2024 POVID [text hallucinations + image perturbations; bimodal negatives] ─┤
      │                                                              ├─ 2025 MMedPO
2025 SIMA [self-generation, self-critique, visual metrics] ────────────┤  [lesion-local counterfactuals
                                                                     │   + medical judge ensembles
                                                                     └─  + bounded clinical weighting]

The underlying progression is from domain instructions → preferences → bimodal counterfactuals → domain-tool weighting. First, the question is whether a model can answer in medical language; next, answer quality is found not to imply evidence fidelity; then image negatives are constructed; finally, the question becomes which errors deserve larger gradients. Attention shifts from average accuracy toward hallucinations, data efficiency, and evidence localization, but MMedPO has yet to clear the last hurdle: validating safety gains with clinical-fact metrics and expert evaluation.

5. Open challenges and actionable research opportunities

5.1 Theory

1. Is tool weighting consistent? With fixed preference pairs and equal training budgets, compare equal-weight DPO, unclipped confidence, [0.5,1.5], [0.75,1.25], and physician-risk-based weights. Measure Spearman correlation between tool confidence and physician-assessed clinical importance, estimating when weighting introduces bias.
2. Counterfactual sufficiency. For the same lesion, compare occlusion, diffusion noise, semantic inpainting, and noise outside its boundary; measure causal effects on answers and lesion facts. If outside-boundary noise changes answers equally, the interpretation that local intervention establishes evidence use fails.
3. Conflict between negative types. At a fixed total budget of 19,170 pairs, sweep text:visual ratios of 0:1, 1:3, 1:1, 3:1, and 1:0, recording per-metric Pareto frontiers to explain Table 8's nonmonotonic -0.10 METEOR effect.

5.2 Engineering and Applications

1. Truly fair competitor comparison. Retrain DPO, POVID, SIMA, FiSAO, and MMedPO with the same LLaVA-Med-1.5-7B, LoRA rank, 19,170 preference pairs, and 4×A6000-hour budget; use at least 5 seeds and paired bootstrap. Table 1 does not establish equal data/compute budgets.
2. Clinical-fact evaluation. On IU-Xray and MIMIC-CXR, report BLEU, RadGraph F1, CheXbert label F1, and lesion-negation error rates together, and ask at least two radiologists to blindly assess 200 pairs. Directly test whether lexical gains translate into factual gains.
3. Cross-tool robustness. Replace MedKLIP with two independent localizers and include human lesion boxes as an upper bound. Compare localization IoU with final factual accuracy to distinguish an effective MMedPO mechanism from a particularly effective MedKLIP tool.
4. End-to-end cost accounting. Report GPT-4o calls, tokens, failed retries, inference GPU-hours for three Med-LLMs, and training GPU-hours. Compare with sampled expert-annotation costs for 19,170 pairs to establish economic value.

5.3 Emerging Directions

1. Multimodal medical evidence chains. Extend a single image to image sequences + laboratory tests + medical history. Compare unified versus modality-decomposed preference weights, testing whether the model flags conflicts between history and images rather than blindly following either modality.
2. Uncertainty and abstention. Have CRS predict both importance and uncertainty, training calibrated abstention on low-consensus samples. Compare selective risk at equal coverage, rather than only overall accuracy.
3. Privacy and bias. Stratify local-perturbation gains by hospital, device, sex, and age, and analyze equalized errors for rare lesions. If MedKLIP localizes poorly in one subgroup, preference optimization may systematically neglect it.

6. Other recent work worth watching

1. VistaDPO(2025.07, ICML 2025) Decomposes video preferences into instance, temporal, and perceptual levels and constructs 7.2K spatial-temporal annotated QA pairs, directly extending evidence localization from medical images to video. PMLR
2. Can We Predict Performance of Large Models across Vision-Language Tasks?(2025.07, ICML 2025) Uses probabilistic matrix factorization and MCMC to predict unmeasured VLM×task performance with uncertainty, reducing multibenchmark evaluation costs without replacing direct tests on critical safety benchmarks. PMLR｜Code
3. LEMUR(ICLR 2026 OpenReview) Uses VLMs for fine-grained, region-level multimodal retrieval; its regional representations may support more reliable ROI selection for MMedPO. OpenReview
4. Robustness of Vision-Language-Action Models against Multimodal Perturbations(ICLR 2026 OpenReview) Extends perturbations to VLA multimodal inputs and actions, providing an adjacent test of whether local-perturbation training transfers to interactive systems. OpenReview
5. mDPO: Conditional and Anchored Preference Optimization(2024–2025 project) Also addresses language preferences overpowering images, using conditional preferences and reward anchors to constrain falling chosen-answer probabilities. An equal-budget, same-backbone comparison with MMedPO is among the most important missing experiments. Project page
6. POVID(2024.02, preprint) Already discussed as a precursor, its public data and two-stage weights make it a practical reproduction baseline: reproduce global perturbations first, then replace them with lesion ROIs. arXiv｜Code / Data / Weights

7. Conclusion

MMedPO's value is not inventing another DPO formula, but making dependence on correct visual evidence a constructible data problem. After stricter comparisons against each column's strongest competitor, Table 1 still leads on 10/10 metrics—consistent directional evidence. Without multiseed runs, clinical-fact metrics, physician evaluation, and tool calibration, however, it is currently best viewed as a design blueprint for medical multimodal post-training rather than definitive proof of clinical reliability.

Reproducibility footnote. All performance figures come from directly read Tables 1–4, Table 8, and the experimental setup of the core paper. No unreported hyperparameter sweeps, standard deviations, or significance results were invented. Index updating was attempted during file delivery; any persistence failure will be stated explicitly in the delivery message.