HEX: Implicit Semi-Autoregressive Experts in Diffusion Language Models

Daily ML/AI Deep Paper Tracking: HEX

Date: 2026-08-17　 Field: LLM / NLP (reasoning and test-time scaling)
Core paper: Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Experts
Authors: Jihoon Lee, Hoyeon Moon, Kevin Zhai, Arun Kumar Chithanar, Anit Kumar Sahu, Soummya Kar, Chul Lee, Souradip Chakraborty, Amrit Singh Bedi
Publication: ICLR 2026 Conference Paper; arXiv:2510.05040v1, 2025-10-06
Positioning in one sentence: A diffusion language model's reasoning capability is not determined by its weights alone. HEX treats “the order in which tokens are unmasked” as a collection of latent semi-autoregressive experts, replacing high-confidence greediness along one path with consensus across block lengths. Additional inference compute thus unlocks capabilities already present but suppressed by poor scheduling.

0. Link verification table

All URLs below were accessed on 2026-08-17. Version dates follow the arXiv submission history. “Not public” means the paper or author website provides no official resource, rather than a claim that no third-party implementation exists anywhere online.

1. Topic selection and candidate pool

The run date is Monday, so the scope is LLM/NLP reasoning, test-time scaling, post-training, and evaluation. The deduplication index shows coverage of EPG, EARL-BO, DuoAttention, SAE canonicalization, and DPLM2.1 within the past 14 days; these do not duplicate this issue's focus on inference decoding schedules for diffusion language models. Candidates came from official ICLR 2026, NeurIPS 2025, and ACL 2026 pages or OpenReview/arXiv, prioritizing main-conference papers and work from the past 12 months, yielding 33 papers. Executed shuf -i 1-33 -n 1 returned 5; materials were complete, with no redraw.

2. In-depth analysis of the core paper

2.1 Motivation: the problem is not “the model cannot do it,” but “the decoding order suppresses capabilities it already has”

Autoregressive language models permit only a left-to-right topology; a masked diffusion language model (MDM) can recover tokens from any masked set. This freedom appears advantageous, but expands the crucial inference-time decision from “what is the next token?” to “which positions should be unmasked next?” The training objective averages over many mask patterns, while natural-language data has a strong sequential bias: prefixes are usually more common and easier to learn from than “knowing the ending and filling the middle.” A uniform masking objective therefore asks a finite-capacity model to solve a combinatorially enormous set of conditional generation subproblems simultaneously, many with almost no identifiable context.

The paper quantifies this mismatch through a counterintuitive failure. Earlier, top-k margin, which chooses decoding positions by confidence, raised Sudoku accuracy from below 7% to nearly 90%; on GSM8K, however, the authors' Figure 2 shows 50.87% for Random versus only 24.72% for top-k margin, and more than 55.5% of top-k-margin runs generate backward from the end of the sequence [AfterEoT], causing catastrophic collapse. Table 1 in the official version more systematically reports the non-semi-autoregressive baseline's GSM8K collapse rate of 55.80% and accuracy of 22.52%; the MATH collapse rate is 29.80% and accuracy of 16.60%. Switching to a left-to-right semi-autoregressive block schedule reduces both collapse rates to 0.00%, with accuracy rising to 76.27% and 32.80%, respectively.

The novelty is therefore not another round of self-consistency, but extending the notion of a “reasoning path” beyond natural-language chains of thought to mask schedule / block size. The same weights implement different conditional distributions at different block lengths, resembling experts without explicitly trained labels. The authors' engineering hypothesis is that schedules make different errors, while correct answers are more likely to recur consistently across schedules; marginalizing over these structural trajectories is more robust than trusting token confidence along a single path.

This matters for three reasons. First, the value of parallel dLLM generation holds only if decoding does not destroy quality. Second, post-training is expensive; if allocating inference compute with fixed weights can outperform GRPO, “training capability” and “eliciting capability” must be studied separately. Third, decoding schedules are a control axis unique to dLLMs, extending test-time scaling beyond sample count and reasoning length to generation topology.

2.2 Narrative: from diagnosing collapse to latent experts and a controllable compute knob

The argument progresses in four steps, rather than simply proposing a method and running benchmarks.

First, the authors formulate the any-order masked-diffusion training objective, showing that it mixes all masking sizes and position combinations in one loss. Training flexibility is not free: the model must learn exponentially many conditional distributions, and language data does not supply equally strong signals for all conditions.

Second, they observe that common confidence-based unmasking underperforms random selection on complex language reasoning. The turning point is [AfterEoT] collapse: high confidence need not help the final answer; it may simply mean “end tokens are easy to predict.” This separates token-level certainty from answer-level correctness.

Third, they restrict scheduling to left-to-right semi-AR blocks. Small blocks provide conditions closer to autoregressive prefixes; large blocks offer more parallelism but sparser context. Different block lengths thus query the same model's conditional predictions over different visible sets. The paper interprets these predictions as “hidden experts” and approximates marginalization over latent experts by sampling across block lengths.

Fourth, instead of a complex verifier, the authors parse final answers and take a majority vote. This trades total FLOPs for reliability while retaining training-free operation and easy parallelism. Table 7 also rules out the simple explanation that “more samples are enough”: a single block length of 32 with 25 seeds achieves 79.53% on GSM8K, while five block lengths × five seeds achieves 88.10%, so structural diversity adds +8.57pp; the corresponding MATH results are 35.80% → 40.00%, +4.20pp.

The implicit tradeoff is clear: HEX does not find the best schedule for each problem, but avoids poor choices through parallel enumeration and voting. Compute redundancy buys a simple, training-free selector. Its claim is closer to “robust ensembling” than “intelligent routing.”

2.3 Formal definitions and mechanism analysis

Let the sequence be (x=(x_1,\ldots,x_n)\in\mathcal V^n), masked positions (M\subseteq[n]), and visible positions (M^c). An MDM learns the conditional denoiser (p_\theta(x[M]\mid x[M^c])). The training objective in Equation (1) can be written as:

[ \theta^*\in\arg\min_\theta \mathbb E_{x\sim\mathcal D}\mathbb E_{\ell\sim\mathrm{Unif}([n])} \mathbb E_{M:|M|=\ell} \left[-\sum_{i\in M}\log p_\theta(x_i\mid x[M^c])\right]. ]

This means the model learns across many visible sets (U) on average. HEX assumes visible context defines a latent expert (p_\theta(x_i=a\mid[x_{prompt},x[U]])). An ideal mixture would enumerate all conditions; in practice, a block-schedule set (\mathcal B) supplies a Monte Carlo approximation (Equation 4):

[ p_{mix}(x_i=a\mid x_{prompt})\approx \mathbb E_{b\sim\mathcal B} \left[p_\theta(x_i=a\mid[x_{prompt},x[U_b]])\right], \quad \hat a=\arg\max_a p_{mix}(x_i=a\mid x_{prompt}). ]

For sequence length (n) and block length (b), the (t)-th contiguous block is:

[ M_t={(t-1)b+1,\ldots,\min(tb,n)},\qquad T(b)=\lceil n/b\rceil. ]

Blocks are completed left to right, while diffusion repeatedly denoises within each block. The system generates complete text for each block length; parser (f(\cdot)) removes LaTeX wrappers, spaces, and commas, then selects the modal answer:

[ \hat y=\operatorname{mode}{f(x_b^{(T)}):b\in\mathcal B, s\in\mathcal S}. ]

Ties favor the answer from the smallest block length. The workflow is:

prompt + [MASK] × L
       │
       ├── block=8   ── diffusion denoise ── parse answer y₁
       ├── block=16  ── diffusion denoise ── parse answer y₂
       ├── block=32  ── diffusion denoise ── parse answer y₃
       ├── block=64  ── diffusion denoise ── parse answer y₄
       └── block=128 ── diffusion denoise ── parse answer y₅
                         × independent seeds
                                  │
                       frequency majority vote
                                  │
                      tie → smallest block answer
                                  ▼
                             final answer

Core pseudocode (a faithful condensation of Algorithm 2):

outputs = []
for block schedule b in B:
    x = [MASK] * L
    for t = 1..T:
        sample predictions for all masked positions from pθ(. | prompt, x)
        fix positions allowed by U[t,b]
        remask the remaining unresolved positions
    outputs.append(parse(x))
return most_frequent(outputs); on tie choose earliest/smallest-block output

The parser is the most consequential engineering detail: majority voting works naturally only in a normalizable answer space. Short mathematical answers and multiple-choice questions fit; open-ended writing first requires semantic equivalence classes, otherwise the string mode underestimates genuine consensus.

2.4 Experimental design, benchmarks, and metrics

All inference methods use LLaDA-8B-Instruct. The main configuration uses 256 output tokens, 128 steps, and 2 unmasked tokens per step. Random chooses two positions uniformly; top-k uses maximum probability; top-k margin selects two positions by the gap between top-1 and top-2 probabilities; d1 is an SFT/RL-trained control. For ARC-C, HEX's authors reproduce d1 with 1 training epoch; GSM8K/MATH use d1's best reported results, so training protocols are not fully uniform across rows.

Complete main-results matrix

The paper reports only four benchmarks; a fifth column cannot be invented merely to satisfy a “five-benchmark” format. All values below come from the official version's Table 3, rows Random through HEX ×5 seeds; all are Accuracy (%). Averages use only reported columns and are derived by this report.

Relative to top-k margin, HEX ×5's absolute/relative improvements are: GSM8K +63.38pp / +256.39%; MATH +23.60pp / +143.90%; ARC-C +33.62pp / +62.05%; TruthfulQA +29.10pp / +102.61%. Relative to Random, the gains are +37.23pp, +23.20pp, +17.75pp, and +15.06pp. Against training-based d1, the three comparable tasks show +8.30pp (+10.40%), +2.80pp (+7.53%), +5.12pp (+6.19%).

The strongest quantitative evidence in the paper is not 88.10% alone, but a three-part chain: top-k margin suffers 55.80% collapse on GSM8K; semi-AR eliminates collapse and raises accuracy to 76.27%; 25-way structured voting across five block lengths raises it further to 88.10%. This closes the loop from failure diagnosis to mechanistic repair to ensemble gains.

2.5 Ablations, sensitivity, phase transitions, and compute cost

Layer-by-layer mechanism ablation

Strictly speaking, the paper does not provide traditional cumulative module ablations within one training pipeline. It provides inference-decoding comparisons corresponding to design levels. The following comes from Tables 1, 3, and 7:

The frequency-versus-likelihood ablation is equally crucial. In Table 3, Random achieves 70.05% on ARC-C, while NLL achieves only 60.84%, lower by 9.21pp; HEX frequency voting on the same candidate set reaches 74.57%, exceeding NLL by 13.73pp. On TruthfulQA, NLL scores 28.07%, even below Random's 42.40% by 14.33pp, whereas HEX reaches 45.91%. Thus, “the model's own likelihood can serve as a verifier” does not hold in these settings.

Sensitivity to the number of dynamic trajectories

Table 2 in the official version (GSM8K) provides a verifiable curve:

Increasing from 5 to 30 trajectories improves accuracy by only 2.19pp (relative +2.67%) and reduces the tie rate by 2.81pp (relative -72.61%). This is clear diminishing marginal return, not continued linear improvement: the main benefit is resolving ambiguity, with additional compute squeezing out a small tail of errors.

Output length and block-length phase transitions

Table 8 reports HEX ×5 at output lengths 128/256/512: GSM8K 81.05/88.10/88.40, MATH 33.40/40.00/47.60, ARC-C 88.40/87.80/87.12, TruthfulQA 53.80/57.46/59.80. Longer outputs clearly benefit mathematics, but ARC-C instead declines from 128 to 512 by 1.28pp, showing that a longer scratchpad is not universally beneficial.

Appendix Figure 10 more closely resembles a “phase transition”: the authors sweep block size in increments of 16 and explicitly report a sharp performance decline in the range 128–256 . The figure does not provide readable exact values for each point, so this report does not guess digitized values. The reliable conclusion is an abrupt drop in the large-block region, not a universally optimal threshold.

Latency and statistical significance

Table 5 in the official version reports single-point latency for HEX ×1 seed (five block lengths, five samples): GSM8K 11.03s versus Random's 2.09s (5.2648×); MATH 13.43s versus 2.56s (5.2461×); ARC-C 11.90s versus 2.27s (5.2473×); TruthfulQA 11.61s versus 2.21s (5.2455×). HEX ×5 seeds uses 25 trajectories, and the paper explicitly states that its total token-level inference cost is approximately this multiple of one top-k run: 25×. Parallelism can reduce wall-clock time but cannot eliminate FLOPs, energy, or memory-throughput costs.

No mean±standard deviation across independent full runs, confidence intervals, or significance tests are reported. “×5 seeds” aggregates 25 samples into one vote; it is not a variance estimate from five independent experiments. Existing figures therefore cannot establish statistical stability across hardware or random batches. A binomial approximation could roughly estimate test-set sampling error, but that is not a reported result, and this report does not present it as evidence of significance.

2.6 Genuine contributions, reusable designs, and transfer potential

The contribution is not majority voting itself, but identifying a third axis of dLLM test-time compute: beyond more samples or longer reasoning chains, one can vary the order of variable revelation. Autoregressive architectures fix token order; MDMs expose it as a control variable. HEX encourages viewing the same weights as a family of behaviors indexed by conditional context sets, rather than a single function.

Three designs are independently reusable. First, monitor trajectory-level failures such as [AfterEoT], repetition, and backward generation before assessing accuracy, avoiding confusion between decoding pathologies and missing model knowledge. Second, distinguish noise diversity from structural diversity; Table 7 shows greater value in the latter, potentially transferring to tree search, speculative decoding, and tool routing. Third, use schedule ensembles as a training-free fallback: when retraining the base model is impossible, a few discrete control parameters can form an inexpensive expert pool.

Transfer opportunities include pixel/patch revelation order in discrete image diffusion, visual evidence before textual conclusions or vice versa in multimodal dLLMs, interface skeletons before local implementations in code generation, and paragraph-block revision in long conversations. The key is not copying the block-length set, but identifying structural families whose visible-variable sets match the task's training-distribution biases.

2.7 Limitations

Limitations stated by the paper

1. Higher inference compute: The official version §6, page 9, explicitly acknowledges additional test-time compute; Appendix B.2 quantifies total token-level cost for ×5 seeds at approximately 25×.
2. Narrow task coverage: §6, page 9, says evaluation focuses on reasoning tasks; open-ended stories, image generation, and long conversations remain untested.
3. Lack of a theoretical explanation: §6, page 9, explicitly states that a theoretical understanding of HEX remains undeveloped; “latent experts” currently serves mainly as an empirical interpretation.

Additional critique (independent observations)

1. Experimental design—“SOTA” depends heavily on compute budget. HEX ×5 uses 25 trajectories, whereas the main table's Random/top-k/top-k-margin each uses one sample. GSM8K 88.10% versus 50.87% is therefore not an equal-FLOPs comparison. It demonstrates effective test-time scaling, not better efficiency per unit compute. A fair experiment would give every method 25 trajectories or identical GPU-seconds.

2. Missing key competitors—no direct equal-budget comparison against self-consistency / verifier search. Table 7 compares only 25 seeds at one block length against 25 samples across five block lengths, without reward models, process verifiers, or adaptive schedule selectors under the same 25× budget. The claim that majority voting is most appropriate remains unresolved.

3. Argument gap—“experts” may simply be slices of conditional distributions. Table 6 only shows that average performance across block lengths is below voting (e.g., at L=256, mean single-expert GSM8K accuracy is 75.39%, versus HEX's 82.18%). It does not measure error-correlation matrices, mutual information, or specialization. The evidence supports ensemble diversity but does not establish identifiable expert structure.

4. Experimental design—d1 comparison protocols are not fully consistent. GSM8K/MATH use the d1 paper's best results, ARC-C uses HEX's authors' 1-epoch reproduction, and TruthfulQA is absent because training data/checkpoints are unavailable. Thus “outperforming GRPO” holds in only three cells, under different post-training budgets and implementations.

5. Metric validity—string-answer consensus is not semantic consensus. The parser removes only LaTeX, spaces, and commas, while TruthfulQA or open text can admit multiple semantically equivalent expressions. Vote frequency can split synonymous answers. The four benchmarks favor normalizable answers, so results cannot transfer directly to the long conversations/stories discussed by the paper.

6. A failing subtask—longer outputs hurt rather than help ARC-C. In Table 8, HEX ×5 declines from 88.40% at L=128 to 87.12% at L=512, while MATH rises from 33.40% to 47.60%. “Monotonic improvement with more inference budget” therefore holds only for Figure 6's increased voting samples, not increased token length.

7. Hyperparameter overfitting—the block set comes from one base model and four benchmarks. The main configuration fixes {8,16,32,64,128}, L=256, temperature=0.9, and 2 tokens per step, using only LLaDA-8B-Instruct. This cannot distinguish a general geometric-progression principle from a sweet spot tuned to one model/length.

8. Insufficient statistical evidence—“significance” lacks repeated-run distributions. The paper reports no mean±standard deviation, confidence intervals, or hypothesis tests, and aggregates five seeds into one prediction. Even the seemingly substantial +2.80pp over d1 on MATH cannot establish stable significance across random inference trajectories.

3. Related-work retrospective: the chain of problems and technical origins

3.1 Self-Consistency — 2023.03, ICLR 2023

Problem solved. Self-Consistency extends CoT from one greedy path to multiple sampled reasoning chains, marginalizing paths by final-answer frequency. Relative to ordinary CoT, it reports gains on GSM8K, SVAMP, AQuA, StrategyQA, and ARC-C of 17.9, 11.0, 12.2, 6.4, and 3.9 percentage points(arXiv abstract). It establishes the basic paradigm of eliciting latent reasoning through answer consensus without changing weights.

Remaining gap. It still assumes autoregressive generation order; paths differ through token sampling. It does not address dLLMs' unique freedom to choose which positions to unmask first, nor separate sampling noise from structural schedule diversity.

How HEX responds. HEX rewrites marginalization over reasoning paths as marginalization over mask schedules. Final-answer majority voting inherits Self-Consistency, but the sample space moves from surface-language trajectories to generation topology.

Design inheritance. “Multiple paths → answer parsing → mode” carries over almost unchanged. HEX adds the block schedule; Table 7 further shows that multiple seeds under one schedule underperform diversity across schedules.

3.2 Large Language Diffusion Models (LLaDA)— 2025.10 v3, ICLR 2025 DeLTa Workshop Oral

Problem solved. LLaDA demonstrates that masked diffusion can scale to 8B through pretraining + SFT, approaching similarly sized autoregressive baselines on general tasks, mathematics, code, and instruction following. Random-mask forward corruption and Transformer conditional denoising provide HEX with a usable base model.

Remaining gap. LLaDA offers any-order generation, but standard inference still requires manually choosing a mask schedule. Model-level flexibility does not answer which order is most reliable for reasoning; confidence heuristics can even trigger end-token collapse.

How HEX responds. HEX fixes all LLaDA-8B-Instruct weights and explicitly explores that freedom with a semi-autoregressive block family, advancing from “can dLLMs generate?” to “how should their learned capabilities be elicited?”

Design inheritance. LLaDA's arbitrary-mask conditional denoiser supplies the shared parameters of latent experts. HEX adds no MoE parameters; different visible sets activate different conditional behaviors during inference.

3.3 Train for the Worst, Plan for the Best — 2025.08 v3, ICML 2025 Outstanding Paper

Problem solved. This work formalizes how MDM training faces numerous difficult infilling subproblems, while adaptive token ordering can bypass difficult conditions during inference. On Sudoku, confidence-adaptive ordering raises accuracy from <7% to roughly 90%, even exceeding an ARM with 7× more parameters (arXiv abstract).

Remaining gap. Sudoku constraint propagation aligns local confidence relatively well with global solvability; in complex language reasoning, end tokens can be highly confident but useless for the answer. This does not explain why the same heuristic reverses from a major Sudoku gain to GSM8K collapse.

How HEX responds. HEX first reproduces that reversal: GSM8K Random 50.87%, top-k margin 24.72%. It then replaces pure confidence selection with prefix-compatible semi-AR constraints, using consensus across schedules to avoid single-schedule risks.

Design inheritance. “Inference order is an optimizable variable” carries over directly; the selection principle evolves from local token margins to answer-level consensus across a family of candidate schedules.

3.4 d1 — 2025.06 v2, NeurIPS 2025

Problem solved. d1 brings reasoning post-training to diffusion LLMs: masked SFT distills reasoning data, followed by online reinforcement learning with critic-free diffu-GRPO. It shows that dLLMs can absorb SFT+RL reasoning gains, providing strong baselines such as GSM8K 79.80% and MATH 37.20% (quoted in HEX Table 3).

Remaining gap. d1 requires data, training, and new checkpoints. It optimizes model parameters but does not systematically exploit inference-time mask order. No corresponding training data/public checkpoint exists for TruthfulQA, so HEX's main table cannot report d1 there.

How HEX responds. HEX takes a complementary route: without parameter updates, 25-way schedule/sample ensembling achieves 88.10/40.00/87.80 on GSM8K/MATH/ARC-C, exceeding d1 by 8.30/2.80/5.12pp.

Design inheritance. d1 attributes capability gains to policy learning; HEX shows that some gains can also come from better elicitation. A natural combination is diffu-GRPO training followed by schedule-ensemble or learned-router inference.

4. Technical evolution

2022–2023  Self-Consistency [AR; multiple sampled CoTs; modal answer]
                 │
                 ├────────── source of consensus aggregation ──────────┐
                 │                                                     │
2025  LLaDA [8B masked diffusion; any-order conditional denoising]      │
       │                                                              │
       ├── Train for the Worst [optimizable token order; margin works on Sudoku]
       │           │                                                  │
       │           └── Question: does local confidence transfer to language reasoning?
       │                                                              │
       └── d1 [masked SFT + diffu-GRPO; training-based reasoning gains] │
                 │                                                     │
                 ├── strong trained baseline                           │
                 ▼                                                     ▼
2026  HEX [semi-AR prefix constraints + multi-block-length hidden experts + answer voting]
       ├─ Repairs AfterEoT collapse
       ├─ Separates structural diversity from seed diversity
       └─ Makes mask schedules a new axis of test-time scaling

The underlying shift is from “weights contain all capability” to “post-training elicitation strategies also matter.” Self-Consistency first shows consensus can refine multiple paths from one AR model; LLaDA adds token-order freedom; Train for the Worst demonstrates the potential of order selection; d1 shows training can strengthen dLLM reasoning; HEX joins these strands into schedule-level self-consistency.

There are two turning points. First, random sampling becomes structural sampling: seeds change noise, whereas block lengths change conditional context. Second, token confidence gives way to answer consensus: local confidence is distorted by [AfterEoT] , while agreement among final answers better matches task metrics. Community attention also shifts from “can dLLMs match AR?” to “how should their non-autoregressive freedom be priced across quality, parallelism, and compute?”

5. Open challenges and actionable research opportunities

5.1 Theory

1. When must majority voting outperform the best single expert? Under a fixed 25-call budget, record error-correlation matrices and conditional accuracy for five block lengths; use correlated Bernoulli ensemble bounds to predict voting gains, then compare predictions with GSM8K/MATH measurements. Average expert accuracy alone, as in Table 6, cannot distinguish genuine complementarity from incidental dispersion.
2. A minimum covering set of mask schedules. At L=256, enumerate or Bayesian-optimize candidate block lengths, comparing {8,16,32,64,128} with equally sized adaptive sets. Minimize total FLOPs while limiting accuracy loss to 1pp. This turns “hidden experts” into a testable set-cover problem.
3. Calibration theory for confidence failures. Build reliability diagrams and ECE measurements for position-level margins, answer NLL, and final correctness; compare Sudoku and GSM8K head-to-head under one evaluation protocol to identify constraint structures in which local confidence is a useful routing signal.

5.2 Engineering and Applications

1. Fair equal-budget controls. With identical 25 forward-equivalent calls, L=256, and LLaDA weights, compare HEX, single-block-length self-consistency, NLL reranking, learned verifiers, and early-stop adaptive consistency. Report accuracy, GPU-seconds, joules, peak memory, and Pareto frontiers.
2. Online early stopping. Use a voting posterior or sequential probability ratio test to stop adding trajectories once consensus reaches a threshold; compare fixed 5/10/15/20/25/30 trajectories. Determine how far average sample count can fall below 25 while maintaining GSM8K accuracy at 88.10%±0.5pp.
3. Cross-base-model reproduction. Run the same block set unchanged on LLaDA-8B, Dream-7B, and at least one adapted diffusion LLM. If optimal block regions vary by model, compare absolute block length with the relative parameterization b/L.
4. Semantic voting for open-ended answers. On long conversations or story continuation, compare string modes, embedding clustering, LLM-judge clustering, and fact-unit voting. Use both human preference and factual consistency to quantify vote splitting caused by parsing.

5.3 Emerging Directions

1. Joint training–inference routing. Starting from a d1 checkpoint, compare “GRPO only,” “HEX only,” “GRPO+HEX,” and “train a per-instance schedule router” on three tasks with identical training data and total inference FLOPs. This directly tests whether post-training and schedule scaling complement or substitute for each other.
2. Multimodal revelation order. In visual question answering, vary the proportions of visual and text tokens unmasked first; compare modality-first, interleaved, and HEX-style multischedule voting. Evaluate answer accuracy and visual grounding, rather than language answers alone.
3. Safety and consensus hallucinations. Construct prompts that elicit common but false claims in training data; compare whether HEX versus NLL/Random causes multiple schedules to form “false consensus” around popular misconceptions. TruthfulQA accuracy of 57.46% still means 42.54% errors; consensus cannot simply be equated with truth.

6. Other recent work worth watching

1. Plan and Budget: Effective and Efficient Test-Time Scaling(ICLR 2026)—Joint planning and budget allocation makes it a useful adaptive early-stop comparison with HEX's fixed 25-way budget. OpenReview
2. Diversity-Aware Policy Optimization for LLM Reasoning(NeurIPS 2025)—Directly optimizes reasoning-trajectory diversity, testing whether structural diversity can also be internalized during training. NeurIPS
3. Enigmata: Scaling Logical Reasoning with Synthetic Verifiable Puzzles(NeurIPS 2025)—Uses verifiable synthetic puzzles to scale logical reasoning, suitable for testing whether HEX works beyond short mathematical answers. NeurIPS
4. GPO: Learning from Critical Steps to Improve LLM Reasoning(NeurIPS 2025)—Focuses on critical-step credit assignment, providing a training-time counterpart to HEX's final-answer-only voting. NeurIPS
5. RFG: Test-Time Scaling for Diffusion LLM Reasoning with Reward-Free Guidance(2025-09-29, arXiv)—Contemporaneous reward-free guidance specifically for dLLMs; a high-priority equal-budget competitor for HEX. arXiv
6. Training Language Models to Reason Efficiently(NeurIPS 2025)—Jointly trains for answer quality and reasoning length, highlighting the need to report efficiency alongside HEX's 25× budget. NeurIPS
7. d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning(NeurIPS 2025)—Scales diffusion reasoning through training, naturally complementing HEX's decoding approach. NeurIPS

7. Conclusion

The key lesson from HEX is not another score increase for an 8B model, but the elevation of generation order to a first-class test-time-scaling variable. High-confidence token selection can fail completely on GSM8K through 55.80% end-token collapse; prefix-style semi-AR constraints eliminate collapse, and structural consensus across block lengths raises 76.27% to 88.10%. dLLM capability evaluation therefore cannot be separated from scheduling.

However, 88.10% is not free: the final configuration requires roughly 25× total inference compute, the main table is not budget-matched, and error bars are absent. The crucial next step is not adding schedules, but comparing adaptive routing, early stopping, verifiers, and schedule ensembles at equal GPU-seconds to establish an accuracy–compute Pareto frontier.

Data provenance and transparency

• All core-paper data comes from the official ICLR 2026 version: Figures 2, 5, 6, 10; Tables 1–8; §5.1, §6, Appendix B.1–B.7. All derived percentage-point and relative-percent changes are calculated directly from the corresponding table values.
• The paper has no fifth main benchmark, standard deviation across independent repeats, or confidence intervals. This report preserves those omissions rather than inventing values.
• Related-work figures come from the respective arXiv abstracts or HEX's citations in Table 3. Third-party implementations are not mislabeled as official code.
• The HTML passed Warm Editorial structural validation (0 warnings). Playwright is callable in the runtime, but neither /usr/bin/chromium, /opt/pw-browsers nor the system PATH contains a Chromium executable, and, as requested, playwright install was not run. Consequently, light/dark/mobile screenshot-based visual review could not be completed for this issue. Responsive table scrolling, mobile SVG horizontal containers, and both theme palettes were inspected in source, which is not equivalent to real-browser screenshot verification.
• The index was updated with this delivery. If remote synchronization fails, the attached report and current index file take precedence.