Transformers are Inherently Succinct

⚠️ Methodological Disclosure

The tracking template requires a five-benchmark × six-method matrix, ablations, and hyperparameter-sensitivity curves. This issue's core paper is purely theoretical, with no experiments, benchmarks, or hyperparameters. Forcing an empirical template would fabricate data. Equivalent substitutions are therefore used and explicitly identified throughout:

Core paper

Versions: v1 = 2025-10-22, v2 = 2025-10-23, v3 = 2026-05-15(this report uses v2). Among subject categories, cs.FL is primary—this is a genuine theoretical computer-science paper.

Related work

2.1 Positioning in One Sentence

The key is not that Transformers are powerful, but the logical connection: because the same concept is compressed so tightly, tools that unfold it for inspection—automata or formulas—explode in size, making verification necessarily EXPSPACE-complete. Succinctness and verifiability are two sides of one coin, the paper's deepest argument.

2.2 Motivation and Problem Evidence

2.2.1 The Field's Bad-News Narrative

Over six years, formal Transformer expressivity research largely followed a path of accumulating negative results:

The natural conclusion is: Transformers, under this idealized model, are less expressive than RNNs, since finite-precision RNNs recognize all regular languages, while star-free languages form a strict subset(classic counterexample: (aa)*, an even number of a's, regular but not star-free).

2.2.2 The Specific Mismatch Identified

The authors note a clear tension with experience: in practice, nobody considers Transformers weaker than equally sized RNNs. The diagnosis is not incorrect conclusions but the wrong measurement dimension:

For a concrete analogy, suppose a concept needs O(n) parameters in a Transformer but 2^(2^n) states in a DFA. Expressivity says both recognize it and are indistinguishable; engineering intuition sees an enormous difference.

This is the novelty: ask not whether a system can, but how large the size gap is under equal expressivity. This had not been asked before: even Yang et al. (2024), despite an exact three-way equivalence, never analyzed the size cost of conversion. Reading the NeurIPS 2024 paper confirms its only size statement appears in Appendix B.2—each attention operation translates into at most 2^(T_A+T_B) operations—and no total size relation across multiple translation steps is given. An exponential blowup sat in an equivalence-proof footnote for two years without follow-up.

2.2.3 Why This Question Matters

1. Theory: succinctness is a mature logical tool (Stockmeyer 1974; Grohe & Schweikardt 2004) never previously introduced into neural expressivity. This is a methodological transfer.
2. Verifiability: size gaps directly determine lower bounds for formal Transformer verification, one of AI safety's major goals. Succinctness is the source of the bad news.
3. Architecture selection: it offers a representation-theoretic explanation for Transformers' practical value, independent of optimization or parallelism: the same knowledge needs fewer parameters.

2.3 The Paper's Argument

The argument has positive and negative sides, reusing one construction twice:

Key turning point

Turning Point A (Engine of the Positive Result): the authors observe that UHAT's strict masking + rightmost tie-breaking provides the primitive of finding the nearest predecessor satisfying a predicate, enabling both counter increments with constraint checks. They infer that Transformers can encode, with polynomial size, a counter of size double exponential, while finite automata must explicitly represent it with doubly exponentially many states.

Turning Point B (The Same Construction in Reverse): the counter construction Meanwhile carries the EXPSPACE-hardness reduction. 2^n-tiling, placing tiles on a 2^n × 2^n grid, is EXPSPACE-complete. Encoding the double-exponential counter enables encoding the grid, so B-RASP non-emptiness is at least EXPSPACE-hard (Prop. 6).

Turning Point C (The Technical Core of the Upper Bound): completeness requires an EXPSPACE upper bound. Naively translating to LTL and using known algorithms yields only 2-EXPSPACE because Yang et al.'s translation is double exponential. The authors use Prop. 11(all values arising during UHAT computation require only polynomial bits) to reduce translation to single exponential(Prop. 12), closing the bounds.

Implicit Design Trade-Off

The paper chooses fixed-precision UHAT, described as the weakest Transformer class in expressive power. This is deliberate self-restriction: It sacrifices direct coverage of softmax / average-hard attention, in exchange for(a) stronger lower-bound significance—even the weakest model is this succinct; (b) fidelity to real fixed-precision implementations; and (c) reuse of Yang et al.'s exact characterization.

2.4 Methodology and Mechanisms

2.4.1 Formal Definitions

UHAT (Masked Unique Hard-Attention Transformer). A masked unique hard-attention layer is a five-tuple:

Fixed precision means exactly that computation always uses real numbers representable with a constant k bits. This distinguishes the setting from log-precision and infinite-precision work.

B-RASP Defines Boolean vectors over input words through two operations: pointwise operations(Boolean combinations of existing vectors at the same position) and attention operations(mask and score predicates select a position, returning its value predicate or a default). A word is accepted when the designated output vector equals 1 at a designated position. Here it serves as an intermediate language between Transformers and LTL, inherited from Yang et al. 2024.

Succinctness Gap — The Core Definition:

Two Easily Overlooked Technical Details:

• Quantifier order: for every f there exists R₁, not vice versa. The gap concerns an asymptotic family, not a single counterexample.
• Fairness of size measures: standard binary encoding is used, but RNN precision k is counted in unary to prevent hiding an exponential by specifying precision 2ⁿ with n binary bits. This detail determines whether Cor. 17 holds(see critique 4 in §2.7).

2.4.2 Key Mechanisms

2.4.3 Core Construction: Double-Exponential Counter (Reconstructing Example 4)

The technical heart: make a UHAT of polynomial size decide whether a string encodes a valid 2ⁿ × 2ⁿ tiling.

3.1 Main Matrix: Succinctness Gaps

Rows are source systems on the Transformer side; columns are compared target systems.

3.2 Complexity Matrix: Verification Problems

This connects two independent research lines: Jerad et al. (2025, ACL) showed tie-breaking direction affects expressivity, with leftmost UHAT corresponding to a strictly weaker LTL fragment; this paper shows it also affects verification complexity. One switch controls both expressivity and verifiability.

3.3 “Ablation”: Assumption-Relaxation Lattice

Theoretical ablation removes premises one by one to see which theorem fails. The table reconstructs proof dependencies (not presented this way by the paper; my reconstruction may be mistaken):

3.4 Quantitative Improvements over Predecessors

This is the most persuasive quantitative table: on one problem, the paper raises the lower bound and lowers the upper bound until they meet. Closing bounds is among the strongest forms of theoretical result, achieved here for all three problems: non-emptiness, equivalence, and universality.

4.1 What Is the Real Contribution?

Not merely proving Transformers more succinct, but: introducing a measure mature in logic for 50 years—succinctness, Stockmeyer 1974—into neural expressivity, and showing it supplies essential explanatory power rather than academic curiosity.

1. Redefining the question: expressivity (whether) → succinctness (how large). The shift makes the puzzling claim that Transformers are weaker than RNNs noncontradictory: weaker by set inclusion, stronger by representation size.
2. Connecting succinctness and verifiability: not two independent results, but two sides of the same double-exponential-counter construction.
3. Closing open complexity gaps: tightening the gap between NEXPTIME-hardness and no upper bound to EXPSPACE-completeness.

Independently Reusable Designs

4.2 Limitations · Category I: Author-Reported

1. Fixed (finite) precision. Defended as faithful to implementation, but excludes log-precision, the standard setting in work by Merrill & Sabharwal and others. Conclusions from the two lines therefore cannot be directly compared.
2. Only unique hard attention. The authors call UHAT the weakest Transformer class and explicitly note that softmax / average-hard attention goes beyond AC⁰ and may recognize more languages.
3. Only star-free languages. A direct consequence of the UHAT characterization.
4. Positioning future work: developing tools to automatically analyze, verify, and explain Transformers despite EXPSPACE-completeness, using symbolic techniques, simulation, and similar methods. This acknowledges that one main result is a negative barrier to circumvent.

4.3 Limitations · Category II: Independent Critique

Critique 1| Succinctness and verifiability results rely on different models, a discontinuity the paper does not flag. (Argument gap)

Thm 14 / 16 counter constructions rely on rightmost tie-breaking, whereas Cor. 13's NEXP result relies on leftmost tie-breaking + strict future masks. Actual evidence is: model A (rightmost) is extremely succinct and EXPSPACE-complete; model B (leftmost + strict future) is in NEXP with unknown succinctness. The narrative that succinctness causes unverifiability suggests an internal tradeoff, but the evidence does not support that causal reading. Establishing it requires proving model B's succinctness gap strictly smaller than A's, which is not done.

Critique 2| Double-exponential succinctness is established by existence of particular examples, not a distributional property. (The theoretical counterpart of experimental-design issues)

The definition uses existential quantifiers, so Thm 16 only guarantees a language family exists, essentially tiling encodings, with double-exponential DFA blowup. It says nothing about prevalence in natural language or practical tasks. Empirically, this resembles reporting infinite gains on one carefully constructed adversarial benchmark without standard benchmarks. No average-case succinctness discussion is offered.

Critique 3| The Comparison with Sälzer et al. Is Not Head-to-Head. (Missing key comparison)

The paper claims to strengthen non-emptiness hardness from NEXPTIME to EXPSPACE. But Sälzer et al. study satisfiability of quantized Transformer encoders (trSAT), finding the general case undecidable and the quantized case NEXPTIME-hard and in NEXPTIME. Model classes and problem definitions differ: their quantized TE has a lower NEXPTIME upper bound, while this paper's UHAT is EXPSPACE-complete, hence higher. Calling it a strengthened lower bound requires comparable model classes, but v2 supplies no rigorous model translation supporting the comparison. That row in §3.4 should be treated as pending verification, not established fact.

Critique 4| Fairness of the RNN Comparison Depends on an Insufficiently Defended Convention. (Metric validity)

Cor. 17, exponential succinctness over RNNs, is a corollary of Prop. 3 (RNN ≡ finite automata) plus Thm 16, not an independent proof. It strongly depends on counting RNN precision k in unary, defended only as preventing unfair comparisons through arbitrary precision. This is a substantive modeling choice: real fp16/fp32 precision is constant, making unary and binary equivalent, but for theoretical RNN precision growing with n, unary counting artificially adds an exponential size penalty. The paper does not disentangle attention's real advantage from this counting convention in Cor. 17's exponential gap. I consider this its weakest conclusion.

Critique 5| The “Succinct ⇒ Hard to Verify” Causal Arrow May Be Reversed, or Both May Share a Cause. (Argument gap)

A more conservative reading is that EXPSPACE-hardness and double-exponential succinctness both arise from UHAT encoding 2ⁿ-tiling—not causation, but a common cause. This matters for future work: under a common cause, succinct yet easily verifiable models may exist if their succinctness does not rely on tiling encodings; under true causation, they cannot. The paper chooses the more memorable formulation over the more rigorous one.

Critique 6| The Practical Meaning of EXPSPACE-Completeness Is Overstated. (Metric validity)

The paper and award publicity describe it as provably intractable, but this is worst-case complexity measured in UHAT description size rather than input length. For small abstract models used in verification, practical feasibility depends on constants and structure, not asymptotic classes; SAT/SMT solvers routinely tackle NP-complete and PSPACE-complete problems. Proposing symbolic workarounds acknowledges that asymptotic classification is not the final word, but the main text lacks that restraint.

Critique 7| No Empirical Sanity Check. (Reproducibility risk)

Pure theory without code, confirmed on the Poster page, is normal in cs.FL. But the central double-exponential-counter construction is implementable: a toy UHAT with dozens of parameters could actually decide 2ⁿ-tiling. Executable validation of theoretical constructions is already routine in mechanistic interpretability, for example Tracr. Without it, Example 4's implementation details cannot be independently checked, although they support three main results: Thm 14 / 16 and Prop. 6.

Critique 8| Changes from v2 to v3 Are Opaque. (Limitation of my analysis)

arXiv lists v3 (2026-05-15), updated after ICLR camera-ready, but its HTML could not be retrieved. All theorem numbering and statements here follow v2; v3 changes could make references inaccurate. This is an explicitly acknowledged limitation.

5.1 Hahn (2020) — Theoretical Limitations of Self-Attention in Neural Sequence Models

TACL 2020 · ACL Anthology · arXiv:1906.06755

① Problem addressed. In the third year of Transformers' NLP dominance, it first established hard impossibility results: hard-attention Transformers cannot recognize PARITY or DYCK-1. A Lipschitz-style sensitivity argument bounds the effect of changing one token, while PARITY requires every bit flip to change the result. In one sentence, it made Transformer expressivity a provable mathematical object rather than intuition.

② Unresolved gap. It provides a collection of negative results rather than an exact characterization. More importantly, it prompted years of misreading: inability to recognize PARITY supposedly means Transformers are weak. Hahn measures only set inclusion, entirely ignoring cost—the blind spot originates here and is exposed only by this issue's core paper.

③ How the core paper responds. It does not overturn Hahn, but changes the coordinate axis. Accepting that UHAT is limited to star-free languages, it shows that within this range, Transformer encoding efficiency far exceeds equally expressive classical systems. Hahn's weakness and this paper's strength are mathematically simultaneously true because they measure different things.

④ Transfer of key designs. Hahn's research paradigm of characterizing neural architectures through formal-language classes is fully inherited; only the comparison relation changes from ⊆ to |·|.

5.2 Barceló, Kozachinskiy, Lin, Podolskii (2024) — Logical Languages Accepted by Transformer Encoders with Hard Attention

ICLR 2024 · proceedings · arXiv:2310.03817

① Problem addressed. Elevates Hahn's scattered negative results to a circuit-complexity classification: UHAT recognizes only languages in AC⁰ , and not all AC⁰ languages; it recognizes all languages defined by first-order logic with arbitrary unary numerical predicates. AHAT, in contrast, can leave AC⁰ for TC⁰ and recognize all FO(All) languages, even with counting terms.

② Unresolved gap.“⊊ AC⁰” is an upper bound rather than an exact characterization: what UHAT equals remains unknown. UHAT versus AHAT also complicates the question: if AHAT is stronger, why do practical softmax models not show corresponding capability jumps? Missing size information resurfaces.

③ How the core paper responds.Inherits its model definitions and UHAT/AHAT hierarchy and deliberately selects the weakest class, UHAT. Because Barceló et al. established this hierarchy, succinctness lower bounds for UHAT automatically extend to stronger classes. Without the hierarchy, “even the weakest class is this succinct” would lose its force.

④ Transfer of key designs. Logical characterization as an analytic tool is inherited. Barceló et al. use FO(All), the core paper LTL; the equivalence of LTL and FO over words, Kamp's theorem, connects them.

5.3 ★ Yang, Chiang, Angluin (2024) — Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages

NeurIPS 2024 · proceedings PDF · arXiv:2310.13897

① Problem addressed. Tightens Barceló et al.'s upper bound into an exact characterization: Transformers with hard attention, attention masking, strict masks, and no positional encoding are exactly equivalent to LTL, defining exactly star-free languages. Technically it introduces B-RASP as an intermediate language and characterizes the expressive effects of positional encoding, strict masking, and depth separately.

② What Remained: Equivalence Without Cost. Reading the NeurIPS text confirms the sole size statement is in Appendix B.2: each attention operation translates into at most 2^(T_A+T_B) operations, an exponential blowup when converting query-and-key score predicates into key-only form. The paper no total size relation across multiple translation steps is given nor discusses efficient compilation among the three systems. This costs twice: Conceptually, readers mistake expressively equivalent for equivalent in every respect, overlooking Transformers' advantage over LTL; technically, algorithms analyzing UHAT through LTL pay an unnecessary exponential factor.

③ The Core Paper Fills the Gap Directly in Both Directions:

• Upper bound: Prop. 11 + 12 reduce UHAT → LTL from double to single exponential, enabling the EXPSPACE rather than 2-EXPSPACE upper bound in Thm 5.
• Lower bound: Thm 14 proves the exponential cannot be eliminated; together with polynomial LTL → UHAT (Prop. 15), this yields an exact directionally asymmetric characterization.

④ Transfer of Key Designs: B-RASP Is Inherited Unchanged as the intermediate language. Thm 5 and Prop. 6 are first proved for B-RASP, then transferred to UHAT. The transfer is: from using B-RASP to prove expressive equivalence to using it for size lower bounds and complexity hardness. The same scaffold serves two purposes.

5.4 Sälzer et al. (2024/2025) — Transformer Encoder Satisfiability: Complexity and Impact on Formal Reasoning

arXiv:2405.18548

① Problem addressed. First systematically asks how hard Transformer verification is. Defines trSAT and proves that for the TE models commonly used in expressivity research, trSAT is undecidable; restricting to fixed-bit-width arithmetic (quantization) makes it decidable, but NEXPTIME-hard, while also supplying a NEXPTIME upper bound for quantized TE. In one sentence, it makes formal Transformer verification a complexity-priced problem rather than a vision, identifying quantization as a decidability switch.

② Unresolved gap. Its decidability switch is quantization / precision, without addressing attention structure such as masks and tie-breaking—the second switch found by the core paper. Nor does it isolate complexity for the precisely characterized UHAT subclass.

③ How the core paper responds. Thm 5 / 18 give completeness for UHAT non-emptiness, equivalence, and universality, closing both bounds. More importantly, Cor. 13 supplies a new complexity-control knob beyond Sälzer's precision axis: attention-mask direction and tie-breaking.

④ Transfer of key designs. The reduction-to-classical-decision-problems methodology is inherited, but the source changes to 2ⁿ-tiling rather than SAT variants, raising complexity to EXPSPACE.

5.5 Jerad, Svete, Li, Cotterell (2025) — Unique Hard Attention: A Tale of Two Sides

ACL 2025 Short, pp. 977–996 · ACL Anthology · arXiv:2503.14615 · DOI 10.18653/v1/2025.acl-short.76

Note: Ryan Cotterell coauthors both this and the core paper, an internal research-group transfer of the problem.

① Problem addressed. Identifies an overlooked implementation detail in Yang et al.'s equivalence: both leftmost and rightmost tie-breaking are allowed. It proves UHAT with leftmost-only tie-breaking corresponds to a strictly weaker LTL fragment; more intriguingly, leftmost hard attention is equivalent to soft attention and may therefore approximate real Transformers better than the rightmost model. In one sentence, tie-breaking direction, normally dismissed as implementation noise, divides expressive power—and the more realistic side is weaker.

② Unresolved gap. It proves tie-breaking affects expressivity, but never asks whether it also affects size / succinctness or verification complexity. Its analysis still uses Yang et al.'s double exponential translation, preventing useful algorithmic upper bounds.

③ Cor. 13 Directly Answers the Second Question: UHAT restricted to strict future masks and leftmost ties has non-emptiness in NEXP . The paper explicitly derives this by improving Jerad et al.'s double-exponential translation, reusing Prop. 11/12's technique a second time.

④ Transfer of key designs. The fine-grained distinction by tie-breaking direction becomes a dimension of complexity classification. Transfer: from which is more expressive to which is more decidable.

6.1 Evolution Tree

6.2 Underlying Logic

Driving Force: Refining the Measure

Each step finds, where the previous question seemed settled, a dimension averaged away by the previous measure. Exact equivalence in Step 3 creates a false sense of closure: the word equivalent is so strong that the community treats the problem as closed for two years.

where the Yang → Sälzer transition is crucial: research moves from characterizing Transformers to using characterizations algorithmically. Once the view becomes algorithmic, size can no longer be ignored because it enters complexity bounds directly. The core paper is a natural consequence of that transition, formally articulated only two years later.

Shift in community attention

Early work (2020–2023) emphasizes capability boundaries to explain Transformer strength; recent work (2024–2026) emphasizes verifiability and implementation details, shifting motivation toward AI safety / formal guarantees. The award itself marks the shift: ICLR names a pure cs.FL theory paper Outstanding, highlighting a negative result.

Reading the Ladder: ✗PARITY (counterexample) → ⊊ AC⁰ (class inclusion) → ≡ star-free (exact equivalence) → NEXPTIME-hard (algorithmic cost) → tie-breaking branches (model refinement) → double-exponential size gap + EXPSPACE-completeness (size and complexity bounds both close). Each level's metric is more quantitative than the last.

8.1 Theory

T1 | Does Succinctness Survive in the Leftmost Fragment?(The most direct gap)

Concrete question: Thm 14/16 use rightmost ties; Cor. 13's NEXP result uses leftmost. With leftmost ties and strict future masks, do UHAT's exponential / double-exponential gaps over LTL / DFA persist?

Why High Value and Low Barrier: if yes, this yields a Pareto-improving architectural constraint: equally succinct but an exponential level cheaper to verify, directly guiding verification-tool design. If no, the causal narrative gains support. Either answer matters, and the required techniques already exist across the two papers.

T2 | Average-Case Succinctness

Concrete question: current existential definitions guarantee only some hard language family. Can a distribution-sensitive succinctness measure be defined and proved, such as expected size ratios over a natural language family?Operational Starting Point: compute exact ratios for natural star-free subclasses, such as piecewise-testable languages, and test whether exponential gaps remain.

T3 | Succinctness Gains from CoT

Concrete question: Merrill & Sabharwal proves CoT increases expressivity. Within the same framework, what is the succinctness gap between UHAT with k CoT steps and UHAT without CoT?

T4 | Cross-Architecture Succinctness Comparison

Concrete question: multiple works compare SSM / Mamba / linear attention and Transformer expressivity, but succinctness is entirely unexplored. Suggested Design: in the same formal-language class, for example star-free, and compare Mamba with UHAT head-to-head by size. This is the missing comparison needed to decide whether linear attention is truly equivalent: current debates concern expressivity, while practical differences may be about size.

8.2 Engineering and Applications

E1 | Make the Construction Executable, Tracr-Style

Concrete question: take Example 4 / Prop. 6's double-exponential counter and actually compile it into weights, verifying 2ⁿ-tiling decisions at small n, such as 3 or 4. Value: (a) independently check the construction underlying three main theorems; (b) provide mechanistic interpretability with a known-ground-truth circuit as a probing benchmark. Barrier: low—Tracr-like tools are mature and the construction is explicit.

E2 | Practical Routes Around EXPSPACE

Concrete question: the authors suggest symbolic techniques and simulation. More concretely, can a parameterized-complexity result be established, for example non-emptiness being FPT in attention depth d? Real small models have single-digit depths. If only depth drives the exponential tower while other dimensions enter polynomially, EXPSPACE may be manageable practically. The asymptotic analysis does not make this decomposition, leaving a clear gap.

E3 | Succinctness as an Architecture-Search Objective

Concrete question: if succinctness explains Transformer advantages, can the minimum size needed to encode a task become an empirical metric？Concrete Form: fix synthetic formal-language families, measure minimum parameters needed for 100% accuracy in each architecture, and plot size versus architecture. This is a direct, feasible translation of theory into an empirical protocol.

8.3 Emerging and Cross-Disciplinary Directions

N1 | Tension Between Succinctness and Interpretability

An unsettling implication is that more succinct models are harder to unfold and inspect. Interpretability often assumes finding the circuit means understanding the model, but a doubly exponentially compressed circuit may be computationally impossible to understand. Operationalization: define explanation size as the size of a human-readable form, such as LTL or a decision tree, translated from UHAT. Thm 14 already gives an exponential lower bound. This is a formal negative result for interpretability that has not yet been explicitly articulated.

N2 | Connections to Memory and Generalization

Another candidate, an ICML 2026 honorable mention, How much can language models memorize? measures capacity at about 3.6 bits per parameter. Succinctness describes the other side: Transformers need fewer parameters for the same information. Can bits per parameter and size ratios relative to DFA be quantitatively connected or derived from each other?Barrier: medium.

N3 | Design Rules for Verifiable Architectures

Cor. 13 shows structural constraints reduce verification complexity by one exponential level. Can complexity-friendly attention constraints be systematically enumerated into design rules? If future regulation requires formal verification for critical models, such rules would connect theory to compliance.

How much can language models memorize? (ICML 2026 Oral / Honorable Mention)

Morris, Sitawarin, Kokhlikyan, Guo, Suh, Rush, Chaudhuri, Mahloujifar. Introduces an information-theoretic capacity measure separating memorization from generalization, estimating about 3.6 bits per parameter. Together with the core paper it forms an information-theoretic dual of the same question(see N2), the most recommended companion reading.
arXiv:2505.24832 🔎 · ICML Oral 🔎 · OpenReview 🔎

To Grok Grokking: Provable Grokking in Ridge Regression (ICML 2026 HM)

Mingyue Xu, Gal Vardi, Itay Safran. Gives a provable characterization of grokking in analytically tractable ridge regression. It reduces a long-observed phenomenon to a closed-form setting, a standard and effective theoretical approach to mysterious behavior.
ICML 2026 award announcement ✅

The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes (ICML 2026 HM)

Taufeeque, Heimersheim, Gleave, Cundy. Uses probes to locate where honesty emerges and disappears during RLVR, a rare mechanistic rather than behavioral alignment study.
ICML 2026 award announcement ✅

Position: The Alignment Community is Unintentionally Building a Censor's Toolkit (ICML 2026 Outstanding Position Paper)

Sarah Ball, Phil Hackemann. ICML 2026's only Outstanding Position Paper argues that alignment control techniques are technically hard to distinguish from censorship tools. Agree or not, it is among this year's most valuable counterarguments.
ICML 2026 award announcement ✅

The Polar Express: Optimal Matrix Sign Methods and their Application to the Muon Algorithm (ICLR 2026 HM)

Amsel, Persson, Musco, Gower. Uses approximation theory to design optimal polynomial approximations for Muon's polar decomposition, specifically for GPUs and low precision. Theory directly reaching engineering is rare, and this is an example.
OpenReview 🔎 · ICLR announcement ✅

LLMs Get Lost In Multi-Turn Conversation (ICLR 2026 Outstanding Paper)

Laban, Hayashi, Zhou, Neville. Scalable multiturn evaluation showing reliability declines sharply under underspecified instructions. Awarded alongside the core paper via a wholly empirical route, illustrating ICLR's equal emphasis on theory and evaluation.
OpenReview 🔎

Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds (arXiv 2026.05)

Explains grokking as competition between memorization and generalization learning rates, moderated by capacity. Together with the two papers above, it forms a coherent triangle worth reading jointly.
arXiv:2605.09724 🔎

Unique Hard Attention: A Tale of Two Sides (ACL 2025 Short)

Discussed in §5.5. If reading only one prerequisite, choose this one: shortest, and directly explains Cor. 13's origins.
ACL Anthology ✅ · arXiv:2503.14615 ✅