Layer Type Is a Hyperparameter, Not a Constant—and It Is Worth 30% Performance

Before the main text, three known limitations are made explicit:

1. Deduplication index unreachable. This was a scheduled cloud run; the local-device bridge never connects during scheduled execution. Therefore ml-report-index.md could not be read, the 14-day duplicate check could not be performed and the index could not be appended automatically.
2. Random selection. 27 AI4Science candidates from 2025-08 to 2026-08; shuf -i 1-27 -n 1 selected index 6, MolGraphBench. Redraws: 0. Material-sufficiency check passed.
3. Analysis uses the full arXiv HTML v3 without supplementary material. Multiple searches failed to locate the cited repository, so hyperparameter search space, per-dataset CKA values, and training times in seconds cannot be verified; every relevant location is marked.

All links were visited. “Not found” means multiple searches failed; “403” means an existing page was blocked by anti-scraping, with title and volume verified through mirrors.

1.1 Focal paper

1.2 Related work

Note: The fetched v3 bibliography lacks Xu et al.'s original GIN paper (2019), while Kipf&Welling, Hamilton, and Veličković are present. This could be incomplete HTML extraction or a real omission. I cannot distinguish them, so flag it without counting it as a critique.

2.1 Positioning in One Sentence

The second clause is derived here; the paper states only the first and misses that the effect is scenario-dependent. This dependence is the most valuable finding the authors themselves overlook (see §2.5.4).

2.2 Motivation and Problem Evidence

The Targeted Pain Point

The literal problem statement is modest: despite growing GNN use for molecular property prediction, a rigorous benchmark remains missing. Behind it lies a more concrete, testable observation. Table 1 categorizes recent “new” GNN architectures by underlying layers:

What This Reveals: Each work chooses a base layer, stacks innovations such as xLSTM, MoE, contrastive learning, KAN, or fingerprint fusion, and attributes all gains to that innovation. None controls for simply swapping the base layer. The subfield's incremental claims rest on an untested assumption: base-layer effects are small enough to treat as background constants.

Quantifying the Pain Point (Calculated Here, Not in the Paper)

Under an identical transfer pipeline—METLIN pretraining, fine-tuning protocol, RT dataset, and MLP head—change only the convolution type:

Why It Matters

• Practical Value: If layer type is worth 30%, every molecular-property project should first sweep four types at almost zero implementation cost.
• Metascientific Value: If layer type is uncontrolled confounding, five years of incremental claims need re-examination: do new modules cause gains, or merely happen to accompany a suitable base layer?
• Theoretical Value: GIN's transfer dominance echoes Xu et al. (2019), where GIN reaches the 1-WL expressivity ceiling. Capacity may be unhelpful when fitting small data from scratch due to overfitting, but pay off when transferring pretrained representations. The paper does not infer this, although the data support it.

2.3 The Paper's Argument

The argument is not a linear method–experiment–result story, but convergence across four criteria:

1. Decompose “best architecture” into four criteria: absolute performance, training efficiency, transfer learning, and prediction quality. The tradeoff is less sharp conclusions for greater robustness. Agreement strengthens a conclusion; disagreement suggests there is no single best architecture.
2. First establish that GNNs merit discussion through GNN-versus-traditional-ML comparisons. If ECFP4+SVM won everywhere, the remaining steps would be pointless.
3. Introduce two orthogonal probes to explain differences: CKA, borrowed from Welsch et al. 2024, measures representation complementarity; high-error molecule analysis measures error distribution rather than magnitude. This is the paper's smartest design.
4. Use transfer learning as the final test, the only criterion addressing representation reusability and the one closest to deployment.

2.4 Methodology and Mechanisms

RDKit parses molecular SMILES into graph G=(V,E) with atoms as nodes, bonds as edges, and node features X ∈ R^{N×118}(periodic-table one-hot). Full pipeline:

The four layer types differ only in AGG / UPD

Key Mechanisms

2.5 Experimental Design and Quantitative Results

2.5.1 Benchmarks and datasets

Five datasets span three application domains and a 6.5× size range:

A Praiseworthy Dataset-Selection Choice: three domains with fundamentally different properties. ESOL/FreeSolv/Lipophilicity are broadly locally additive; B3DB depends on global pharmacophores; RT concerns chromatographic interactions. This enables §2.5.3's finding, although the authors do not exploit it.

2.5.2 Main Results Matrix (Table 2 · RMSE ± SD · n=3)

SOTA attribution: B3DB ← GMC-MPNN (Nguyen 2026); ESOL/FreeSolv/Lipophilicity ← MolGraph-xLSTM (Sun 2025); RT ← GNN-TL (Yang 2021). The RT SOTA value has a serious issue; see §2.7 ①.

Derived Quantity 1 · Mean Cross-Dataset Rank (1 = Best of 12 Methods)

• GraphSAGE has the best mean rank, 3.00, but greatest range, 1→6; GCN is worse at 4.00 yet stable at 3/3/3/3/8. The paper concludes GCN and GIN are best—inconsistent with mean ranks(GIN sixth, GAT seventh). This clearly reflects weighting across four criteria, including fastest training for GCN/GIN, but the weighting is unstated, making a constructed conclusion look directly read from data.
• SVM ranks first on B3DB, the only traditional ML method beating all eight GNN variants on any dataset.
• LR's RT RMSE is 601.600, 6.7× GraphSAGE's. This is large enough to question whether LR was properly standardized. An obviously failing baseline inflates every other method's relative advantage.

Derived Quantity 2 · The Actual Gap to SOTA

The abstract says its GNNs outperform or match SOTA on three datasets. Literally true, but the omitted part matters more:

2.5.3 Fingerprint Fusion (Derived Matrix Not in the Paper)

The paper concludes fingerprints and graph embeddings are inherently noncomplementary, with no gains except FreeSolv. I calculated all 4 architectures × 5 datasets = 20 paired gains in Table 2; negative means improvement:

Testable Hypothesis: ECFP4 encodes substructure presence, akin to pharmacophore information. For global-pharmacophore targets such as permeability or chromatographic interactions, or very small datasets, it supplies priors GNNs cannot learn from limited data. For locally additive physicochemical quantities such as solubility or logD, message passing already learns efficiently, and sparse 1024-dimensional fingerprints introduce noise and overfitting risk.

The paper compresses this structured, explanatory phenomenon into “noncomplementary”—its most severe loss of information.

2.5.4 Transfer Learning and the Transition (Table 4 · MAE · RT)

Pretraining: METLIN, 80k SMILES with reversed-phase RT. Fine-tuning: 1,400 HILIC RT records. Three granularities:

Why Does GIN Win in Transfer? The paper offers no explanation. Consistent with Xu et al. (2019), injective sum aggregation reaches the 1-WL ceiling and preserves rich structure. This capacity is a liability on small datasets from scratch—GIN ranks tenth on B3DB—but can be exploited during 80k-molecule pretraining. A cheap falsification test: pretrain at 10k/40k/80k METLIN molecules and test whether GIN's relative advantage grows monotonically.

2.5.5 Statistical Significance (Calculated Here, Not in the Paper)

The paper reports n=3 standard deviations but never performs a significance test while repeatedly claiming a best architecture. Two-sample t tests (df=4, α=0.05, t_crit=2.78) compare best versus runner-up architectures per dataset:

Transfer Results Are Not Affected in the Same Way: GIN 52.914±1.218 vs. GraphSAGE 70.752±2.750: the 17.8 gap exceeds eight pooled standard deviations. Again, the robust discovery lies in transfer, while the abstract emphasizes fragile from-scratch findings.

2.5.6 Prediction Quality: High-Error Molecules (Table 3, %)

1. SVM's 18.868% high-error rate on B3DB substantially beats every GNN; the best GNN, GAT at 25.157%, is 33.3% higher. Coupled with SVM's best RMSE of 0.471, traditional ECFP4+SVM defeats all eight GNN variants on B3DB under two criteria. Yet the conclusion broadly claims GNNs outperform fingerprint-trained ML, directly falsified on one fifth of datasets and dismissed in text as “except for B3DB.” B3DB is the sole biological dataset; this valuable counterexample is treated as noise.
2. Table 3 Has No GNN-FP Rows. Prediction quality is one of four criteria, and noncomplementarity a central conclusion, yet fusion-model quality is unevaluated. GNN-FP could leave RMSE unchanged while reducing high-error tails. This missing experiment costs essentially nothing because models are already trained.

2.6 Assessment of Value

Substantive Contributions

not“Running a GNN benchmark”—such studies appear every year. Yes: elevating base-layer type from background constant to measurable variable and showing a 30% effect in transfer learning. The strength lies in obtaining it under fully controlled conditions: identical data, pretraining corpus, fine-tuning, and regression head, changing only AGG/UPD.

Independently Reusable Designs

1. The High-Error-Molecule Analysis Paradigm. Moves evaluation from average error magnitude to which molecules incur errors. Comparing RDKit descriptors across high/low-error subsets reveals significantly different ring counts in 3/5 datasets, an actionable applicability boundary. Independent of this paper's conclusions, this transfers directly to molecular-regression diagnostics.
2. CKA as a Preliminary Probe for Fusion. Measuring complementarity before deciding to fuse is a compute-saving habit, even though low CKA does not guarantee useful fusion—a valuable negative result itself.
3. Three-Level Fine-Tuning Protocol. More granular than frozen versus fully tuned, separating representation usability from adaptability. Table 4 shows GIN's smallest head-to-full gain, 13.2%, indicates the highest-quality pretrained representation: a linear head alone reaches 60.960, better than other architectures fully tuned. This diagnostic logic is reusable.
4. Cross-Domain Dataset Selection(physicochemical / biological / analytical chemistry) enables §2.5.3's domain-dependence finding. Future fusion and complementarity studies should inherit this three-domain setup.

Implications for Future Work

• Scenario-Dependent Architecture Sensitivity. Architecture importance may vary with pretraining scale. Study pretrained molecule count versus inter-architecture spread, predicted to increase monotonically. If confirmed, it quantitatively guides when architecture tuning matters.
• Measuring Representation Complementarity. Low CKA without fusion gains shows similarity metrics cannot predict benefits. A new measure should ask whether differing information relates to labels—conditional, not marginal, complementarity.
• Meta-Evaluation of Benchmarks. This attempts the stricter protocols urged by Bechler-Speicher et al., yet repeats the statistical weaknesses it criticizes. The recursion shows: benchmark papers themselves need benchmarks.

2.7 Limitations

Category I: Author-Reported Limitations (Conclusion)

1. Failure on large molecules:“Unlike ML models that generate fixed-size fingerprints, GNNs predict large complex molecules less accurately.” Figure 4 supports this through significant ring-count differences between high/low-error groups on 3/5 datasets.
2. Training cost:“GNN training takes substantially longer than ML models, potentially limiting scalability in resource-constrained environments.” (Figure 3)
3. Fingerprint noncomplementarity:“Fingerprints and graph embeddings are inherently noncomplementary, with gains only on FreeSolv.” Section 2.5.3 argues that this claimed limitation is itself incorrect.
4. Transfer domain shift (§3.5):“METLIN uses reversed-phase chromatography, whose distribution is entirely different [from HILIC].”

Category II: Eight Independent Critiques

① [Argument Gap / Factual Error] RT's SOTA Baseline Contradicts the Paper's Own Text, Invalidating Comparisons.

Table 2 lists RT SOTA as MAE = 70.40, attributed to GNN-TL (Yang et al., 2021). Elsewhere the text states:

The same paper gives 70.40 and 38.6 for one baseline, an 82% difference. The choice reverses conclusions:

The abstract cites GIN's RT MAE 63.783 among three SOTA achievements—if 38.6 is correct, that achievement is actually a 65.2% deficit.

② [Statistical Validity] Best-Architecture Claims Lack Significance on 4/5 Datasets, with Systematically Low Variance Estimates.

As §2.5.5 shows, only B3DB's best–runner-up gap is significant (t=3.19 > 2.78); other t values are 0.29–1.54. Yet GCN/GIN are called optimal. Further, LR/SVM SD=0.000 shows all runs share a split, so SD measures initialization only. Even the sole significant B3DB result rests on understated uncertainty.

③ [Argument Gap] The Data Refute Fingerprint Noncomplementarity and Reveal an Overlooked Pattern.

Fusion improves 10/20 cells, stratified by dataset: B3DB 4/4, RT 3/4, FreeSolv 3/4 benefit; ESOL and Lipophilicity each 0/4. Calling it noncomplementary except on FreeSolv ignores consistent B3DB/RT gains and loses the explanatory pattern of benefits on biological/analytical endpoints and harm on physicochemical endpoints.

④ [Argument Gap] Low CKA but Ineffective Fusion Is Labeled Counterintuitive and Abandoned Despite Testable Mechanisms.

At least three explanations are testable, but none is tested:

• Fusion-method hypothesis: Concatenation plus a two-layer MLP may fail to exploit complementarity. Compare gated fusion, cross-attention, or a fingerprint branch predicting only residuals.
• Capacity / dimension imbalance: F_embed dimensions are undisclosed; a much wider branch may dominate. Sweep the F_embed/G_embed dimension ratio.
• Conditional-complementarity hypothesis: CKA measures marginal representation differences, while fusion benefits depend on differences conditional on labels. Representations may differ greatly in information irrelevant to the target.

⑤ [Design Confound] Sum Pooling for GIN versus Mean for Others Makes Architecture and Pooling Effects Inseparable.

The paper acknowledges this when explaining low CKA between GIN and others: it may arise from add versus mean pooling. The confound affects not only CKA, but every GIN conclusion, including the key transfer result. For GIN 52.914 versus GCN 75.621, the 30% gap cannot be attributed separately to injective aggregation and sum pooling. Sum pooling retains graph size, and retention time strongly correlates with molecular size, a highly plausible alternative explanation. Repair is cheap: GIN+mean and GCN+add, 2×5=10 extra runs. The Most Necessary Missing Ablation.

⑥ [Terminology] Calling Regression-Head Fine-Tuning Zero-Shot Learning.

The paper says “Even with zero-shot learning (fine-tuning MLP regression head), MAE of 60.960 was achieved.” Updating a head on target data is standard linear probing, not zero-shot. This creates false expectations for later citations. True zero-shot—direct METLIN-pretrained inference on HILIC—is not reported, although it is the key measurement for direct transfer across chromatography modes.

⑦ [Coverage Gap] Prediction Quality Is Never Evaluated for GNN-FP.

Table 3 has only eight rows: four ML and four pure GNN, omitting all fusion variants. Yet noncomplementarity is central. Avoiding catastrophic predictions can matter more than mean accuracy in drug discovery; these four near-zero-cost rows are missing.

⑧ [Reproducibility] The Declared Repository Cannot Be Located, and the Title Changes Across Versions.

v3 says data and code are available at github.com/rajanbit/MolGraphBench . Exact-name, site-specific, and author-associated searches all failed to locate the repository while other repositories by the author were found. v2 offered materials only on reasonable request. Combined with the title change, the repository may be unreleased or defunct. Search grids, dataset optima, F_embed dimensions, and training times exist only in code/supplements, so the paper is currently effectively unreproducible.

3.1 Neural Message Passing for Quantum Chemistry — 2017.04 · ICML 2017 Main Track

Problem addressed. Before MPNN, Duvenaud's neural fingerprints, Kearnes' graph convolutions, and Li's GGNN were isolated and hard to compare. Gilmer et al. unify them as message function M, update function U, and readout R. In one sentence, it turns molecular GNNs from isolated tricks into an architecture family with explicit design dimensions. Its best ensemble reached SOTA on all 13 QM9 targets, 11/13 at chemical accuracy.

What gap remained? MPNN defines a design space but gives no guidance on choosing a point within it. Several M/U/R variants are tested only on QM9's idealized small-organic-molecule quantum chemistry. The gap: does the optimum move when tasks become experimentally measured solubility, permeability, or retention time? MPNN never answered, and few systematically asked over the following decade.

The Core Paper's Response. MolGraphBench directly targets this: fix everything except M/U—118-dimensional atom features, FC projection, GraphNorm and residuals, two-layer head—and vary four instances across five experimentally measured datasets. This is the experiment MPNN could have done but did not.

Transfer of key designs. The three-part abstraction becomes the control scaffold: separable M/U/R makes changing only layer type technically feasible. But one dimension is lost: MPNN also treats readout R as variable; here mean pooling is fixed except GIN's sum. That incomplete control causes §2.7 ⑤'s confound.

3.2 MoleculeNet: A Benchmark for Molecular Machine Learning — 2018.01 · Chemical Science

Problem addressed. Before 2018, molecular ML used incompatible datasets and splits. MoleculeNet unified 17 datasets, over 700,000 compounds, and 800 tasks with DeepChem implementations and recommended splits. ESOL (1,128), FreeSolv (643), and Lipophilicity (4,200) originate there. In one sentence, it gave molecular machine learning common measurement standards.

What gap remained? MoleculeNet standardized data but inadvertently entrenched an evaluation culture equating value with RMSE reductions in standard tables. Architecture, hyperparameters, and splits disappear into a method black box. This leads directly to Table 1's many new architectures claiming gains without component attribution.

The Core Paper's Response. MolGraphBench inherits data and splits but changes evaluation's Dependent variable: instead of asking which method has lowest RMSE, ask how much one controlled design dimension matters. Adding biological B3DB and analytical RT enables §2.5.3's domain-dependent finding.

Transfer of key designs. Benchmark evolves from a dataset collection into controlled experimental design. MoleculeNet supplies the horizontal dataset axis; MolGraphBench attempts the controlled-architecture vertical axis.

3.3 FP-GNN — 2022.11 · Briefings in Bioinformatics

Problem addressed. Graphs and fingerprints offer different strengths: GNNs learn task-dependent continuous representations, while fingerprints encode expert substructure priors. FP-GNN uses GAT for graphs and an ANN for three complementary fingerprints (MACCS + PubChem + Pharmacophore ErG), fusing at a fully connected layer. In one sentence, it demonstrates complementary graph and fingerprint signals, with fusion beating either alone. Best on 7 of 16 tasks over 13 benchmarks; LIT-PCBA mean AUC 0.739; mean AUC 0.849 over 14 breast-cancer cell-line datasets, exceeding AttentiveFP, GAT, GCN, MPNN, and XGBoost.

What gap remained? It establishes that fusion works, but not why it works or when it works. It changes three things together—fingerprint branch, three fingerprints, GAT—preventing attribution, and never tests fingerprint count or consistency across domains.

The Core Paper's Response. GNN-FP revisits this paradigm under controls, extending fusion from GAT to all four layers across five datasets, the attribution experiment FP-GNN omitted. But the conclusion that fusion fails does not hold because this paper uses only ECFP4 (1024 bits), whereas FP-GNN's central claim rests on three complementary fingerprints. It mistakes falsifying a weaker proposition for refuting a stronger one.

Transfer of key designs. Fusion location transfers intact: fully connected fusion becomes post-pooling concatenation plus two MLP layers, essentially equivalent. Fingerprint diversity is what is lost. The domain-dependence finding actually adds support for FP-GNN: fingerprints help all 4/4 architectures on biological B3DB, matching FP-GNN's biological-activity focus.

3.4 GNN-TL — 2021.09 · Journal of Chromatography A

Problem addressed. HILIC retention labels are costly and public datasets contain only thousands of records. GNN-TL first pretrains on ~306K in-silico pseudo-labeled HILIC retention examples then fine-tunes on small real HILIC data. In one sentence, large-scale in-domain pseudo-label pretraining breaks the data bottleneck for small-sample chromatography prediction. The 1,400-molecule RT dataset and its repository come from this work.

What gap remained? Using only one architecture proves transfer helps this task but leaves pretraining gains inseparable from choosing the right architecture. In-domain synthetic HILIC pretraining also leaves open whether only out-of-domain reversed-phase data would suffice.

The Core Paper's Response. Table 4 answers both more strongly than the authors recognize:

• First, rerunning the pipeline with four architectures yields a 30.0% spread—transfer gains are deeply entangled with architecture, making GNN-TL's single-architecture conclusion incomplete.
• Second, out-of-domain METLIN pretraining still gives GIN 52.914 MAE versus 63.783 from scratch, a 17.0% improvement—transfer across chromatography modes works, a valuable extension of GNN-TL.

Transfer and Breaks in Key Designs. Three-level fine-tuning improves the methodology, but a serious bookkeeping error occurs: GNN-TL is 70.40 MAE in Table 2 and 38.6 in text. With in-domain 306K versus out-of-domain 80K pretraining, direct comparison was already inappropriate; claiming to surpass a comparable SOTA is the weakest link in the inheritance chain.

3.5 Position: Graph Learning Will Lose Relevance Due to Poor Benchmarks — 2025.02 · ICML 2025 Position Track

Problem addressed. A diagnosis, not a method. Bechler-Speicher and 11 coauthors, including Bronstein, Morris, Perozzi, and Galkin, argue poor benchmarks erode relevance: excessive molecular-graph focus neglects high-impact combinatorial optimization and chip design; datasets lack representativeness; protocols fragment; single-metric accuracy emphasis promotes benchmark overfitting rather than generalization. In one sentence, it identifies benchmark quality itself as the bottleneck to graph-learning progress. Three proposals: meaningful benchmarks, stricter evaluation, and closer domain-expert collaboration.

What gap remained? As a position paper, it diagnoses without an executable prescription: how many repeats, which splits, significance tests, and confound controls are left to the community.

The Core Paper's Response. MolGraphBench cites it and concretely responds by moving from reporting one score to controlling variables and measuring effects across four criteria.

4.1 Evolution Tree

4.2 Internal Logic of the Evolution

Driving Force: From Can It Learn, to What It Learns, to What Is Doing the Learning

• 2017–2018 (Can it learn?): The question was whether neural networks could beat handcrafted molecular descriptors. MPNN and MoleculeNet answered yes and built infrastructure. Architecture itself constituted the entire innovation.
• 2021–2025 (What does it learn?): With infrastructure in place, innovation shifted to added priors: large pseudo-labeled data (GNN-TL), expert fingerprints (FP-GNN), sequence modeling and MoE (MolGraph-xLSTM), and interpretation tools (CKA). Architecture moved from innovation object to experimental background; four layer types became arbitrary foundations. This is when the variable was lost.
• 2025–2026 (What is doing the learning?): The position paper reflects doubt about how much of five years' progress is real. MolGraphBench first quantifies that doubt, asking not how good its model is, but whether earlier gains were correctly attributed.

Key Transitions Between Stages

First, in 2018: after MoleculeNet, success shifted from working on private data to lower numbers in common tables. A major advance, at the cost of collapsing multidimensional choices into one leaderboard coordinate.

Second, ongoing in 2025–2026: from optimizing leaderboard position to auditing leaderboard validity. MolGraphBench, the position paper, and new frameworks such as LeMat-GenBench, AtomBench, and NewtonBench jointly mark this shift. The shift is meta-level: research methods themselves become the object of study.

Trajectory of Community Attention

Accuracy (2017–2020) → data efficiency and pretraining (2020–2023) → interpretability and representation analysis (2023–2025) → attribution validity and evaluation protocols (2025– )

MolGraphBench sits at the beginning of the last stage. Its weak statistics, uncontrolled confounds, and bookkeeping errors typify an early phase: awareness of the problem has arrived; methodological discipline has not caught up.

6.1 Theory

① Quantifying Expressivity versus Transfer Gains. Not vaguely studying expressivity, but asking: is there a measurable relationship between GIN's 1-WL ceiling and transfer advantage? Fix RT downstream and pretrain four architectures on METLIN at {5k,10k,20k,40k,80k,160k}; plot pretraining scale against performance spread. Falsifiable Prediction: Under the expressivity hypothesis, GIN's relative advantage grows monotonically; a plateau suggests another source, such as graph-size information retained by sum pooling. High value, low barrier—about 1.5× the paper's total experimental compute.

② Construct a Conditional-Complementarity Measure. CKA measures representations'marginal differences, while fusion depends on conditional on labels differences. A new measure should estimate CI(A,B;y) = I(B;y | A) —how much information about y B adds given A. Estimate conditional mutual information with MINE or InfoNCE and test whether it predicts the sign of gains across the 20 cells. If successful, it becomes a prescreening tool for multimodal molecular models.

③ Isolate Pooling Effects. A 2×4 factorial design: {mean,sum} × {GCN,GAT,GIN,GraphSAGE} across five datasets. The sole purpose is disentangling §2.7 ⑤'s confound. 10–40 runs make this paper's—and other GIN-based work's—claims attributable.

6.2 Engineering and Applications

④ Domain-Aware Fingerprint Fusion. Section 2.5.3 suggests an immediately usable draft rule: enable fusion for biological endpoints such as permeability, toxicity, and activity, and chromatographic/analytical endpoints; disable it for locally additive physicochemical endpoints such as solubility, logD, and free energy. Validate 4 architectures × {plain,+FP} across all 17 MoleculeNet datasets, stratified by domain. If domain reliably predicts benefit, use it as an engineering default.

⑤ Ablate Fusion Mechanisms. Fix ECFP4 and compare concatenation, gating, cross-attention, and residual fusion, where fingerprints predict only GNN residuals. Crucial Control: strictly balance F_embed and G_embed dimensions. Expected Result: If residual fusion reverses ESOL's +23.76% degradation into improvement, the problem is fusion, not complementarity itself.

⑥ Minimum Acceptable Evaluation Standards. The community needs specifics: (a) at least 5-fold CV × 3 seeds = 15 runs, decomposing split and initialization variance; (b) test statistics for every superiority claim; (c) equal, disclosed hyperparameter-search budgets for all baselines; (d) cited SOTA split, pretraining data, and metric definitions. With (a) and (b), this paper would not have made its architecture claims on 4/5 datasets.

⑦ Locate the Mechanism of Large-Molecule Failure. Ring counts differ between high/low-error groups on 3/5 datasets, but this is correlation. Create controlled molecule pairs with increasing ring counts and matched other descriptors, measuring error curves crossed with receptive field, or depth L. Testable Hypothesis: If receptive fields are insufficient, 5–6 layers with virtual nodes or global attention should selectively repair large-molecule errors without affecting small molecules.

6.3 Emerging Directions

⑧ Automate Architecture Selection. The paper advocates tuning layer type but manually sweeps four options. Next, search AGG/UPD families with differentiable NAS and compare against the cheap four-layer sweep head-to-head under equal compute. Clear Gap: No one knows whether a learned aggregation function actually beats selecting the best of four known choices for molecular regression. That comparison is the decisive missing evidence for whether NAS is worthwhile.

⑨ Cross-Chromatography / Cross-Assay Transfer Benchmarks. Reversed-phase → HILIC transfer incidentally works here, improving GIN 17.0%. Formalize a transfer matrix across modes and protocols, relating domain distance to transfer gains, with direct metabolomics-annotation value.

⑩ A Reproducibility Contract for Benchmark Papers. The repository problem is not unique. A concrete proposal: arXiv benchmark papers should support continuous automated repository-accessibility checks with latest-check timestamps and status displayed on the page. This is a workflow gap, not a technical challenge.

Other unselected AI4Science candidates worth following.

LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Models (2025.12, v2 2026.01)

A materials-domain counterpart: crystal-generation evaluation is fragmented across stability, novelty, and uniqueness definitions. Read alongside MolGraphBench for evaluation-protocol auditing.
arXiv

NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents (2025.10)

Moves scientific-law discovery beyond toy symbolic regression toward generalization, confronting how to distinguish memorized laws from genuinely discovered ones.
arXiv · HTML

Six Open Questions in Machine-Learned Interatomic Potential Foundation Models (2026.06)

A position paper on MLIP foundation models, analogous to the graph-learning position paper, useful for quickly mapping the subfield's problems.
arXiv · HTML

Can LLM Design High-Quality Experiments? (OptED) (2026.08)

The newest candidate here. Autonomous experimental design comes closest to closed-loop science, yet serious benchmarks remain scarce.
arXiv · HTML

Agentic Self-Driving Microscopy Benchmarks Support Qualification but Do Not Necessarily Generalize to Unseen Tasks (2026.08)

The title states a negative conclusion, a kind of honest agent benchmark result that is too rare. Best paired with OptED.
arXiv · HTML

Open Materials Generation with Inference-Time Reinforcement Learning (OMatG) (2026.02, v2 2026.06)

Introduces inference-time RL into crystal generation, a clear example of test-time scaling extending beyond LLMs to scientific generators.
arXiv · HTML

PoET-2: Understanding protein function with a multimodal retrieval-augmented foundation model (2025.08, NeurIPS 2025 poster)

Retrieval augmentation and multimodality for protein language models; the sole NeurIPS main-track-level protein candidate here.
arXiv

With Five Minutes: Read Table 4. Under an identical transfer pipeline, four GNN layer types span MAE 75.621 to 52.914, a 30.0% gap from changing the convolution. That number should change your first step in the next molecular-property pipeline.

When Citing It: Cite Table 4 and the transfer-learning conclusion. Do not cite the abstract's three-dataset SOTA claim: the real record is one win, one tie, two losses, and one contradictory baseline. Nor Do not cite fingerprint noncomplementarity: its own data show improvements in 10/20 cells, stratified by domain.

When Building on It: Start with the pooling factorial experiment (§6.1③). Until GIN and sum pooling are separated, the main conclusion cannot be attributed.

These quantities were recalculated from Tables 2/3/4 and are not reported by the paper: