In-Depth Report: Grokking in Neural Networks as a Dimensional Phase Transition

I. Research background

Problem and importance

Neural-network training exhibits a puzzling phenomenon: models first reach 100% training accuracy but poor test performance, a memorization phase; later, despite unchanged training loss, test accuracy suddenly rises toward generalization. This abrupt transition from memorization to generalization is called grokking.

Grokking matters for three reasons:

• Foundations of learning theory: Challenges classical monotonic assumptions about generalization/overfitting, suggesting nontrivial learning dynamics.
• Practical applications: Understanding when/why generalization occurs informs model design, tuning, and early stopping.
• Theoretical depth: Connects feature learning, implicit bias, and phase transitions, providing a natural experimental platform.

Standard settings and evaluation

Grokking is usually studied in:

Classic experimental frameworks: Modular arithmetic, such as a + b mod p, and student-teacher settings. These simple, controlled tasks have clear memorization→generalization stages for detailed dynamics analysis.

Standard metrics include:

• Memorization-stage accuracy(training–test accuracy gap)
• Grokking delay(epochs/time between perfect training fit and substantial test improvement)
• Phase-transition point(transition onset: when accuracy begins rising)
• Generalization-window width(duration of the grokking process)

Frontier disagreements and directions

Several theoretical explanations offer insights without unified consensus:

• Circuit-formation perspective: Grokking corresponds to forming compact, interpretable computational circuits, drawing on mechanistic interpretability.
• Feature-learning perspective: Evolution from random features toward meaningful structure gradually yields generalizable representations.
• Phase-transition perspective: Borrowing physics, grokking is a qualitative system change at a parameter threshold—the core paper's framework.
• Information-theoretic perspective: Focuses on compression, redundancy removal, and minimum description length.

Among these, Phase-transition perspective has drawn attention by connecting to mature frameworks such as statistical physics and Singular Learning Theory (SLT), enabling deeper theoretical predictions.

II. Core paper deep dive

Paper overview

Title: Grokking as Dimensional Phase Transition in Neural Networks
Source: arXiv 2604.04655 (April 2026)
Core contribution: Finite-size scaling reveals that grokking corresponds to effective gradient-space dimensionality crossing from sub-diffusive (D < 1) to super-diffusive (D > 1) behavior, clarifying the geometric mechanism behind the transition.

Motivation and narrative

Existing explanations focus on representations or network structure, leaving why grokking looks qualitatively similar across model topologies insufficiently answered. This motivates the hypothesis that grokking reflects not architecture-specific behavior but an intrinsic geometric property of gradient dynamics.

The authors quantify gradient-vector-field geometry in parameter space using effective dimension—how fully gradients occupy high-dimensional space—to track qualitative changes during training.

Methodological details

3.1 Defining and computing effective dimension

Given gradients $\{\nabla_\theta L(\theta(t), x_i) : i=1,\ldots,N\}$ at time $t$, with $N$ training examples, effective dimension is defined as:

D(t) = (Σ λ_i)² / Σ λ_i²

Here $\lambda_i$ are gradient-covariance eigenvalues. Derived from the Rao–Cramér bound, this measures gradient spread in parameter space.

• When $D \ll 1$ (sub-diffusive), gradients concentrate in a low-dimensional subspace and learning follows narrow directions.
• When $D \gg 1$ (super-diffusive), gradients spread more uniformly, enabling learning along multiple independent directions and richer representations.

3.2 Experimental design

Setting: Finite-size scaling uses modular arithmetic ($a + b \bmod p$) at 8 model sizes, with 50+ independent runs per size for robust statistics.

Key controls: To separate gradient-covariance geometry from backpropagation-specific correlations, compare:

• Real training: Standard gradient descent with full backpropagation information.
• Synthetic Gaussian gradients: Replace actual gradients with samples from matching-covariance Gaussian distributions, preserving dimensional information while removing backpropagation correlations.

Real gradients show clear $D$ excess, while synthetic gradients remain at $D \approx 1$, indicating that backpropagation correlations drive the dimensional crossing.

3.3 Connection to Self-Organized Criticality (SOC)

The authors connect observations to statistical-physics SOC, where systems spontaneously approach criticality with long-range correlations and power laws. They argue:

Gradient-dimensional crossing signals evolution toward criticality. Near that point, sensitivity to parameter perturbations peaks, matching rapid test-accuracy changes during grokking.

Main results and numbers

Quantitative features: Across all 8 tested models, the transition when $D$ crosses $D=1$ and the first substantial test-accuracy increase, such as 5%→50%, closely coincide (Pearson $r > 0.95$), strongly supporting dimensional transition as a grokking trigger.

Narrative summary

The argument connects problem, insight, and method as follows:

1. Observation: Similar grokking across topologies suggests a universal geometric principle.
2. Hypothesis: Gradient dimensionality in parameter space is central to that principle.
3. Tool: Finite-size scaling and effective dimension $D(t)$ quantify gradient geometry.
4. Validation: Experiments across scales/topologies link dimensional transitions to grokking.
5. Mechanism: Controls identify backpropagation correlations as the source of excess dimension, suggesting intrinsic SOC behavior.
6. Theoretical significance: Recasts grokking as a physical phase transition, opening a new theoretical perspective.

III. Related work

Technical evolution

Research on grokking and phase transitions follows intertwined threads:

Thread 1: Discovery and initial explanations (2022–2023)

Key work: Power et al. (2022), “Grokking: Generalization beyond Overfitting on Small Algorithmic Datasets”

This pioneering work first systematically observed and named grokking. On simple modular arithmetic, models fit training data before a large test-accuracy jump, a delayed-generalization phenomenon previously little studied systematically. Power et al. contributed by:

• Quantifying generalization delays and their dependence on dataset size and weight decay.
• Ruling out explanations such as learning-rate schedules and batch effects, identifying intrinsic learning dynamics.
• Establishing an experimental foundation for subsequent theory.

Four major explanatory threads followed:

(1a) Circuit formation(Nanda et al., 2023, etc.) emphasizes efficient computational subgraphs forming during grokking; their formation time explains delayed generalization.

(1b) Representation learning(Liu et al., 2022, etc.) sees gradual evolution from random features to structured representations: high-dimensional noisy features during memorization, compressed meaningful low-dimensional representations during generalization.

(1c) Efficiency learning(Varma et al., 2023) studies gradual improvement in encoding efficiency, representing the same data with fewer parameters.

Thread 2: Physical phase-transition perspectives (2023–2024)

Key work 2a: Rubin et al. (2024), “Grokking as a First Order Phase Transition in Two Layer Networks” (ICLR 2024)

A milestone for this perspective, Rubin and colleagues use adaptive kernels—a recent feature-learning theory tool—to rigorously model two-layer-network grokking as a physical first-order phase transition.

Core findings:

• At grokking, internal representations jump from coexisting inefficient mixed states to a single efficient ordered state.
• The jump exhibits first-order characteristics: finite-size supercooling/hysteresis and convergence of transition points across scales.
• The relationship between transition point and weight-decay strength can be quantitatively predicted.

Rubin supplies rigorous mathematical foundations, but largely for special two-layer tasks; generality remains limited.

Key work 2b: Watkins et al. (2024), “Grokking as a Phase Transition between Competing Basins: a Singular Learning Theory Approach”

This work uses Singular Learning Theory (SLT), developed by Watanabe and others for complex models such as neural networks, to revisit grokking.

SLT's free energy generalizes AIC/BIC. Analyzing its topological changes, Watkins et al. show:

• Grokking corresponds to a transition between competing basins on a free-energy surface.
• A singularity at the transition qualitatively changes the learning coefficient.
• This explains sudden generalization changes at identical training loss.

Thread 3: Self-organized criticality and gradient dynamics (2023–2025)

Key work 3: Ghavasieh et al. (2024–2025), “Self-organized criticality in neural networks,” and related work

This thread introduces self-organized criticality (SOC) from studies of sandpiles, earthquakes, brain dynamics, and other complex systems. SOC predicts spontaneous evolution toward scale-invariant, power-law critical states.

In neural-network training, SOC appears as:

• Gradient covariance spontaneously developing special scale-invariant distributions.
• Critical states making dynamics especially sensitive, with small perturbations producing large responses—a physical basis for rapid learning during grokking.
• SOC complements Rubin's phase transitions: the former emphasizes dynamics, the latter final-state properties.

Thread 4: Gradient dimension and geometric transitions (2026)

The core paper and contemporaneous work

By introducing effective dimension as a quantitative tool, 2026 work including arXiv 2604.04655 offers a unified geometric view:

• Gradient-dimensional transitions are indeed geometric manifestations of phase transitions.
• Dimensional transitions and SOC's move toward criticality are physically equivalent: at criticality, gradients maximally fill high-dimensional space.
• Dimensional transitions can serve as geometric indicators of feature-learning phases: sub-diffusion corresponds to low-dimensional noisy features, super-diffusion to high-dimensional meaningful features.

Unresolved questions in this lineage

Despite these insights, key questions remain:

1. Limits of universality: Most theory is validated on simple modular arithmetic. Does the grokking–dimension link hold for image classification and language modeling?
2. Multiple scales: Do different layers' gradient dimensions transition independently, and how are they causally related?
3. Control and acceleration: If dimensional transitions drive grokking, can direct intervention accelerate or control it?
4. Interaction with implicit bias: Gradient descent tends toward simple solutions; is dimensional transition a cause or consequence of that bias?

IV. Structured critical review

This section adds independent assessment of the paper's value and limitations.

1️⃣ Omitted metrics and selective measurement

Question: The paper reports $D(t)$ over time and alignment with test-accuracy transitions. But are key metrics omitted?？

My analysis:

• Well-reported metrics: Test accuracy, gradient-dimension trajectories, transition times.
• Missing but useful metrics:
  • Dimension-dependent sample complexity: Grokking-delay changes with dataset size $N$ are not systematically reported. Power et al. (2022) already found power-law dependence on $N$, but this new explanation does not clarify whether that relationship changes.
  • Separating weight norm from weight decay: Could dimensional growth be an apparent geometric change from shrinking weight norms? Synthetic-Gaussian controls partly address this, but do not fully control weight-decay strength $\lambda$.
  • Smoothness of validation performance: The focus is sudden accuracy jumps, yet grokking also features changes in acceleration(curvature) over the transition. Can dimensional changes explain this second-order property?

Conclusion: Metric selection is reasonable, but linking dimensional transitions to data complexity and quantifying transition smoothness need work. “Dimensional transitions explain grokking” should specify that the explanation addresses when it occurs, with limited predictive power over how gradual the transition is.

2️⃣ Hyperparameters versus tuning-free claims

Question: Are extra hyperparameters introduced, and are settings shared or dataset-specific?

My analysis:

The covariance-derived definition of $D(t)$ is parameter-free, but its interpretation uses thresholds:

• Definition: $D=1$ defines the phase boundary. Why exactly 1 rather than 0.5 or 1.5? The paper insufficiently justifies this choice. Physical critical points often follow symmetry, whereas this one appears empirical.
• Gradient-covariance estimation: Finite-sample covariance estimates are noisy. How many samples estimate $\Sigma$, and is that fixed or dependent on $N$?

Conclusion: Parameter-free effective dimension is an advantage, but justification for $D=1$ is more retrospective observation than prior theoretical derivation. This limits prediction: observed phenomena can be explained afterward, but new models/tasks still require empirical validation.

3️⃣ Are improvement sources isolated?

Question: Does the central “dimensional transition causes generalization” claim reflect a new mechanism or merely more detailed covariance statistics?

My analysis:

The control design is strong:

• Experiment A (real gradients): Standard training preserves backpropagation information; $D$ substantially exceeds 1 and aligns with grokking.
• Experiment B (synthetic Gaussian gradients): Random Gaussian replacements preserve dimensional information while removing backpropagation correlations; $D \approx 1$ and no grokking occurs.

This clearly indicates that excess dimension arises from backpropagation structure rather than gradient space's natural high dimensionality. Thus the gain is identifying a new mechanism, not merely refining statistics.

Potential omission: Experiment A does not report whether stochastic versus full gradients or different batch sizes change transition timing/magnitude, raising the effect of estimation noise on dimensional measurement.

Conclusion: The core relation—dimension excess ⇔ backpropagation structure—is well isolated, but robustness across gradient estimators remains untested.

4️⃣ Curves rather than single points

Question: Are full $D(t)$ trajectories reported, and does transition time scale predictably with system size?

My analysis:

An advantage is reporting complete $D(t)$ trajectories rather than single points, exposing transition smoothness and excess-growth rates.

Finite-size scaling covers 8 model sizes, but the relationship between transition time $t_c$ and parameter count $m$ is not explicitly reported, such as $t_c \propto m^\alpha$. This matters for predicting/controlling delay.

Conclusion: Curves are welcome, but quantitative cross-scale scaling laws are missing, limiting predictive power.

5️⃣ Failure modes and boundary cases

Question: When does the claim fail? Are counterexamples reported?

My analysis:

Every reported experiment aligns dimension changes with grokking, but many realistic conditions could break this correspondence:

• Strong regularization: Large weight decay $\lambda$ could suppress dimensional growth despite meaningful representation learning.
• Noisy data: Noisy or mislabeled datasets may never reach $D>1$; this is untested.
• Extreme overparameterization: When parameters greatly exceed samples, gradient dimension might approach parameter count from the start, requiring a redefined transition.
• Natural-data generalization: Only simple modular-arithmetic tasks are tested. Are transitions clear on CIFAR-10 or ImageNet?

Conclusion: Failure conditions and boundaries are not systematically explored, limiting theoretical completeness and practicality.

6️⃣ Do conclusions exceed the evidence?

Question: Does the scope of the fundamental-explanation claim match experiments?

My analysis:

Claims have several levels:

• Level 1 (fact):“On modular arithmetic, gradient-dimensional transitions closely coincide with abrupt test-accuracy rises.” ✅ Well established.
• Level 2 (causality):“Dimensional transitions cause or drive grokking.” ⚠️ Partly established. Synthetic controls link excess to backpropagation, but direct interventions manipulating dimension to accelerate grokking are needed.
• Level 3 (universality):“Dimensional transitions explain every grokking phenomenon regardless of task/topology.” ❌ Limited evidence, restricted to small modular-arithmetic problems.
• Level 4 (physical universality):“This reveals deep links between deep learning and statistical-physics phase transitions.” ⚠️ More speculative than empirical; SOC/SLT links are conceptual rather than rigorous derivations.

The paper's The title/abstract lean toward broad Level 3–4 claims, while evidence mainly supports Levels 1–2, risking overgeneralization of a setting-specific finding.

Conclusion: The paper should distinguish demonstrated from conjectured claims more carefully; current wording suggests stronger universality than warranted.

7️⃣ Reproducibility

Question: Are code, weights, data, and configurations public?

My analysis: The arXiv abstract page gives no explicit code-release address. Modular arithmetic is deterministic, but covariance estimation/eigenvalue ordering involve numerical details whose implementations may differ subtly.

Conclusion: Full code/data release after acceptance is strongly recommended.

Review summary

V. Links

Core paper

Related prior work (ranked by academic impact)

Additional resources

• Interactive grokking visualization:Google PAIR - Grokking
• Grokking blog overview:Eric Michaud's blog
• Introduction to Singular Learning Theory:LessWrong - Distilling SLT
• Mechanistic interpretability and circuit exploration:MechRL - arXiv 2605.26343

VI. Conclusions and next directions

Value and positioning

The core paper, arXiv 2604.04655, introduces effective gradient-space dimension as a diagnostic for a new geometric perspective on grokking. Its main value:

1. Conceptual innovation: Reframes grokking from a representation-learning stage to a phase transition in gradient geometry, opening a new analytical dimension.
2. Cross-topology generality: Finite-size scaling shows consistent qualitative dimensional transitions across topologies, suggesting underlying universal mathematics.
3. Causal isolation: Synthetic-gradient controls identify backpropagation structure rather than data/architecture as the source of excess dimension.

However, its current applicability is limited: all validation uses small modular-arithmetic tasks. Whether dimension changes drive grokking in practical image classification or language-model training remains open.

Key unresolved questions

Questions deserving immediate attention:

Recommended research directions by difficulty

Short term (3–6 months):

• Test robustness under OOD data and noisy labels.
• Expand to larger synthetic tasks such as high-dimensional linear regression, varying difficulty.
• Release full code/data for independent reproduction and extensions.

Medium term (6–18 months):

• Validate on medium-scale image classification such as CIFAR-10, comparing representation rank and loss-landscape curvature.
• Develop and test dimension-based regularization or early stopping.

Long term (18+ months):

• Establish rigorous correspondence between gradient dimension and SLT free energy.
• Track dimensions during large-model pretraining and relate them to emerging in-context learning or chain-of-thought capabilities.

Final comment

This work advances theoretical deep learning from describing phenomena toward mechanisms. Gradient dimension supplies a physically interpretable diagnostic, but its real strength depends on retaining explanatory power when moving from toy problems to realistic tasks. Evidence supports depth in small controlled settings; universality still requires substantial empirical work.