Circuit Insights: From Activations to Weights and Circuits

4.1 Transcoders and two kinds of attribution

Let the target transcoder feature at layer l be (l,i), with encoder vector f_enc^(l,i); at an earlier layer l' , feature (l',i') has decoder vector f_dec^(l',i')and activation at token t given by a_(l',i')[t]. The simplest cross-feature attribution is:

[ A_{(l',i')\rightarrow(l,i)}(t)=a_{(l',i')}(t),\langle f_{dec}^{(l',i')},f_{enc}^{(l,i)}\rangle. ]

The first factor varies with input; the second depends only on weights. The paper further introduces a local Jacobian:

[ A^{J}{(l',i')\rightarrow(l,i)}(t)=a{(l',i')}(t),f_{dec}^{(l',i')}J^{(l\rightarrow l')}(t)f_{enc}^{(l,i)}, ]

where J Attention, normalization, and nonlinearities are locally frozen at the given input (Eq.2–3, PDF p.3). This lets CircuitLens trace paths in a particular run, but also makes it a local linear attribution rather than a globally causally equivalent account.

The contribution of attention head (l',h) from source token s to target token t to the feature is written as:

[ A_{h,s\rightarrow i,t}=\operatorname{score}{l',h}(r{pre}[t],r_{pre}[s])\cdot\langle (W_{OV}^{l',h})^Tf_{enc}^{l,i},r_{pre}^{l'}[s]\rangle. ]

On the output side, the target feature’s logit contribution to predicted token y_t is computed as:

[ A_{i\rightarrow y_t}=a_{l,i}[t], f_{dec}^{l,i}J^{y\rightarrow(l,i)}[t]W_U[:,y_t]. ]

The first three equations respectively identify fixed connections, the activation path for this input, and the direction of output influence.

4.2 WeightLens pseudocode

for layer l from early to late:
  for feature i in transcoder(l):
    input_tokens  = zscore_outliers(W_emb · f_enc[l,i], τ_vocab)
    output_tokens = zscore_outliers(f_dec[l,i] · W_U, τ_vocab)
    upstream = union over l'<l of
               zscore_outliers(W_dec[l'] · f_enc[l,i], τ_feature)
    input_tokens += inherit_validated_tokens(upstream)
    validated = {token | forward(token) activates feature(l,i)}
    if validated is empty: mark context-dependent
    else: lemmatize(validated) [+ optional output_tokens / LLM refinement]

Vocabulary outlier threshold: for GPT-2, z=4; for Gemma-2-2B and Llama-3.2-1B, z=4.5; cross-feature connection threshold z=3(PDF pp.4–5). The core assumption is that if fixed connections describe intrinsic token semantics, isolated-token forward validation should still activate the feature.

4.3 CircuitLens pseudocode

cache max activations over corpus
partition activation distribution into B=20 quantile bins
sample N=100 without replacement with weight w_i = 1 / n_bin^0.9
for each sampled context:
  trace significant upstream features and (head, token, relative-position)
  mask non-contributing input tokens
  optionally trace affected generated tokens
  form sparse contributor set S_j
filter contributors appearing in at least ρ fraction of inputs
compute Jaccard(S_a,S_b)
cluster; explain each cluster with an LLM; merge cluster descriptions

The main text says DBSCAN (PDF p.6), while Appendix E.5 says HDBSCAN,eps=0.7、min_samples=3(PDF p.21). This naming inconsistency must be clarified for reproduction.

7.1 Five layers × six methods: Clarity

The table selects five early, transition, and late layers from paper Table 3 for a six-method comparison on the same benchmark;bold marks each row’s best value。top uses the 2.3B-token large set; other CircuitLens variants use 24M tokens, while MaxAct* also uses the large set.

Source: Circuit Insights PDF p.25, Table 3. All methods perform poorly at layer 12, consistent with strong context dependence. The large-corpus top variant leads, showing that circuit methods have not eliminated coverage limitations.

7.2 Equal-weight means over ten layers: absolute and relative changes

Sources and calculation: Tables 2/3, PDF pp.24–25; arithmetic means over 10 layers computed by this report. MaxAct* differs between the tables because Table 2 compares only token-based features verifiable by WeightLens, whereas Table 3 uses the CircuitLens setup. They cannot be treated as one population across tables.

The central quantitative findings are not a single SOTA claim:

1. WeightLens Clarity rises from 0.499 to 0.634, an absolute +0.135and relative +27.05%; however, Purity falls from 0.670 to 0.536, an absolute −0.134and relative −20.00%. Clearer does not mean more complete.
2. With24M tokens, WL+CL-Full Clarity is 0.506, above MaxAct*’s0.429 using2.3B tokens, an absolute +0.077, using 98.96%fewer corpus tokens. Yet its Responsiveness/Purity are lower by 0.119/0.131, so less-data superiority cannot be generalized to every dimension.
3. With the same 2.3B tokens, CL-Full(top) exceeds MaxAct* in Clarity/Responsiveness/Purity by 0.189/0.071/0.020, corresponding to relative gains of 44.06%/10.17%/3.23%; Faithfulness is lower by 0.003。
4. Faithfulness is very low for all methods, mostly0.02–0.06. The clearest descriptions do not produce strong output control.

8.1 WeightLens component ablations

Table 2 gives three combinations. WL alone has ten-layer mean Clarity of 0.634; adding output tokens lowers layer 0 from 0.67 to 0.63 and layer 12 from 0.47 to 0.38, with declines at nearly every layer. LLM summarization only partly recovers the loss. Unverified unembedding outliers thus resemble noise rather than free functional semantics. Responsiveness improves slightly for +Out , but Purity and Faithfulness do not improve systematically. The effective component is fixed connections plus single-token forward validation, not adding more words to descriptions.

8.2 CircuitLens component ablations

On small data, ten-layer mean Clarity for CL-Input and CL-Full is respectively 0.429/0.429, so output tracing adds almost no mean benefit; combining WeightLens raises it to 0.506. This contrasts sharply with Table 1’s costs:15-token output analysis adds 12 GPU-hours without improving average Clarity. In the large-data top setting, CL-Full’s Clarity exceeds CL-Input by 0.618 vs 0.609(computed by this report), a small gain. Output tracing is therefore better suited to qualitative functional localization than a default exhaustive step.

8.3 Layer-depth phase transition

The fraction verifiable by WeightLens in Gemma/Llama declines from early to middle layers, then partially recovers late. Although Gemma layer 12 has l0=6and is very sparse, it is hardest to describe using fixed tokens, while early layers with l0=70–88 show abundant token-level structure (PDF pp.7–8). Sparsity is thus insufficient for interpretability. CircuitLens cluster counts also vary by layer: layer 0 averages 1.05 clusters, and layer 7 4.5、layer 12 2.8、layer 21 2.95(Figure 10, PDF p.10). Layer 7 appears to mark a transition from token detection to composite circuits.

8.4 Hyperparameter sensitivity

The paper gives only defaults:B=20, α=0.9, N=100, eps=0.7, min_samples=3, with qualitative discussion rather than systematic sweeps. Appendix E.5 acknowledges that clustering splits monosemantic features into subclusters such as white clothing and green/black bags; sometimes one cluster is clearer.not reported by the paper ρ, z thresholds, andeps/min_samples lack complete grids, stability intervals, or performance curves. Default robustness across models is therefore unproven, and 4.5 clusters cannot be confidently interpreted as true polysemy rather than oversplitting.

8.5 Statistical significance

Main tables report layerwise means, butNot reportedstandard deviations, confidence intervals, random seeds, bootstrap estimates, or significance tests. Table 2 uses roughly 250 features per layer without sampling variance. Small0.01–0.03 differences should not determine reliable winners; the 0.189 Clarity gap is larger and worth replicating but still needs independent repeats. By contrast, PRISM Table 1 reports 95% CIs—for example GPT-2 XL neuron PRISM(max) AUROC 0.85, CI0.78–0.91 (PDF p.6)—showing that comparable work can report uncertainty.

12.1 Transcoders Find Interpretable LLM Feature Circuits（Dunefsky et al., NeurIPS 2024）

Problem addressed: Replaces dense MLPs with sparse transcoders, compares against SAEs on 120M,410M, and 1.4B-parameter models, and decomposes cross-layer edges into activation and fixed-weight factors (arXiv v2, Abstract; PDF p.1).
Remaining gap: Primarily manual cases and the GPT-2 greater-than circuit; the authors acknowledge that transcoders approximate and lose information, while compute limits restrict hyperparameter exploration (PDF pp.10–11, Limitations).
How the core paper responds: WeightLens automates fixed-connection case studies and adds single-token validation; CircuitLens extends prompt-specific attribution into a sampling, clustering, and description pipeline.
Transferred design: activation × decoder–encoder dot product Directly enters core Eq.1; separating input-dependent and invariant factors is the paper’s algebraic foundation.

12.2 Circuit Tracing（Ameisen et al., 2025）

Problem addressed: Uses a cross-layer transcoder replacement model to generate a single-prompt attribution DAG, with error nodes making replacement outputs locally match the original model.
Remaining gap: The project page explicitly notes that replacement mechanisms may differ from original-model mechanisms; attention-dominated tasks can invalidate graphs, and error nodes constitute dark matter.
How the core paper responds: Adopts Jacobian corrections and attention-head token attribution, transforming single-prompt graphs into sparse contributor sets across examples.
Transferred design: Inherits local linearization through frozen attention/normalization, feature-attribution graphs, and awareness of errors and redundancy; however, error-node sufficiency is not a primary metric in the core paper.

12.3 FADE（Puri et al., ACL Findings 2025）

Problem addressed: Divides good descriptions into four diagnostic dimensions: Clarity, Responsiveness, Purity, and Faithfulness. PDF pp.3–5 specify Gini, natural-example classification, and steering procedures.
Remaining gap: Evaluation depends on synthetic examples, an LLM judge, and feature interventions; the paper estimates full experiments at approximately 4,920 GPU-hour（FADE PDF Appendix，p.17）。
How the core paper responds: Adopts FADE in full rather than reporting only a simulated-activation score, with particular discussion of low Faithfulness.
Transferred design: The four metrics define Tables 2/3’s coordinate system, exposing the clearer-but-less-pure tradeoff.

12.4 PRISM（Kopf et al., 2025）

Problem addressed: Rather than forcing one label per feature, clusters highly activating sentence embeddings into 5 groups and describes each. On GPT-2 XL neurons, PRISM(max) AUROC is 0.85 (0.78–0.91), above MaxAct’s 0.53 (0.49–0.58)（Table 1, PDF p.6）。
Remaining gap: Clustering uses semantic embeddings rather than computation paths. The mean variant sometimes substantially underperforms max, revealing a coverage–precision tradeoff.
How the core paper responds: Replaces sentence embeddings with upstream features and attention-head patterns, attempting to split polysemy by mechanism rather than topic.
Transferred design: The cluster-description→merged-description structure transfers directly to CircuitLens, with a crucial change in clustering criterion.

12.5 Output-Centric Feature Descriptions（Gur-Arieh et al., ACL 2025 Main）

Problem addressed: Uses post-steering token changes or direct unembedding projections to describe output influence; evaluation covers 2,560 MLP features, with human evaluation sampling100 Gemma SAE features (PDF pp.6,8).
Remaining gap: Input- and output-side descriptions often disagree; unembedding supplies only a direct linear projection and misses indirect cross-layer effects.
How the core paper responds: WeightLens inherits f_dec·W_U, while CircuitLens adds Jacobian attribution over generated tokens.
Transferred design: Two-sided trigger–consequence descriptions become CircuitLens-Full; ablations nevertheless show that output analysis does not always improve average Clarity.

12.6 Nonlinear technical evolution tree

                    Automatic interpretation: highly activating examples → LLM labels
                         /                    \\
        SAE: sparse features from polysemantic neurons    Output-Centric: output consequences
             |                    |                    |
      FADE: four metrics          PRISM: semantic clusters |
             |                    /                    /
              |                  /                    /
       Transcoder: reconstruct MLP mappings; separate input dependence/fixed weights
                     |                         |
          Circuit Tracing: single-prompt DAG    Weight-connection case studies
                     |                         /
                      |                       /
              Circuit Insights (this paper)
               ├─ WeightLens: fixed weights + token validation
               └─ CircuitLens: path masking + mechanistic clustering + output attribution
                        |
             Next: edge interventions, statistical stability, formal certificates

Three shifts drive this evolution: from whether a neuron resembles a concept to whether sparse features are more monosemantic; from whether descriptions resemble activating examples to whether they explain output consequences; and from semantic similarity between sentences to shared computation paths. This paper occupies the third shift but has not reached formal verification or counterfactual causal certificates.

12.7 Cross-paper comparison

Theory

1. Local attribution versus true causal edges: Compare CircuitLens Jacobians, Integrated Gradients, activation patching, and exact edge ablation on the same feature edges. Preregister edge-level precision/recall, output KL, and task-performance recovery. If Jacobian edges cannot predict patching effects, calling them circuits is unwarranted.
2. Clustering stability: Bootstrap the corpus for each feature; compare HDBSCAN circuit clusters with PRISM semantic clusters using adjusted Rand index, cluster-count variance, and human concept agreement. Sweep eps∈[0.3,0.9]、min_samples∈{2,3,5,10}to locate stable regimes.
3. Identifiability: Build semi-synthetic Transformers with known ground-truth circuits and test whether WeightLens/CircuitLens descriptions remain invariant under functionally equivalent basis changes. This distinguishes mechanistic invariants from coordinate artifacts.

Engineering and applications

4. Equal-budget comparison: Fix20 A100-hours and compare24M CircuitLens,2.3B MaxAct*, PRISM, and output-centric methods on explanation coverage×FADE×blinded human evaluation. Unequal data and compute budgets in the paper cannot determine the best production choice.
5. Selective output tracing: Compare tracing 15 tokens for every feature with tracing 5 tokens only for late-layer/low-Clarity features. Report total GPU-hours, Clarity, Faithfulness, and cost per0.01 gain to test whether most of the additional12 GPU-hours can be eliminated.
6. Cross-domain transfer: Generate descriptions for the same Gemma transcoder on Pile, code, medicine, and Chinese, then evaluate a4×4 cross-domain matrix. If cross-domain Purity collapses, weight-based explanations have not escaped data context.

Emerging directions

7. Safety-circuit monitoring: Compare activation-only, CircuitLens, and formal circuit verification on jailbreak/refusal pairs. Evaluate dangerous-edge recall, false positives, post-attack transfer, and real-time latency. The goal is a verifiable guardrail, not attractive retrospective labels.
8. Training data→circuit formation: Map influential training examples found by Mechanistic Data Attribution to CircuitLens clusters; remove/add very few examples and test whether circuits appear or disappear as predicted, establishing a data–circuit–behavior causal chain.
9. Agentic automatic explainers: Compare AgenticInterpBench’s hypothesis–validate–explain agent with a single gpt-4o-mini under equal token/tool budgets. Evaluate component accuracy on unseen circuits, validation-failure rate, and human-review time.