Structure Tokens: The Problem Is Supervision Geometry, Not Just Compression
Daily ML/AI Paper Review · 2026-08-16
ICML 2025 SpotlightSunday: AI4ScienceRandom selection from 28 candidates
Bit-level supervision, geometric bias, and structural representation alignment let DPLM-2.1's 650M model reduce PDB time-split RMSD from 5.307 to 2.370. CAMEO results, physical validity, and incomplete reproducibility still define clear limits.
Date: 2026-08-16 Field: AI4Science / protein foundation models Core paper: Elucidating the
Design Space of Multimodal Protein Language Models (ICML 2025
Spotlight)
Positioning in one sentence: The problem is not just information loss when tokenizing 3D structures; geometrically similar structures become unrelated categorical labels, teaching the language model the wrong supervision geometry. Bit-level supervision, geometric inductive biases, and representation alignment reconnect discrete protein models with continuous 3D space.
0. Link verification table
All links were visited on 2026-08-16. Not public/not found means no separate entry was verified in the paper, project, venue, or official repository. arXiv dates refer to specific versions.
Reachability Note. Science returned 403 during automated retrieval, but DOI, PubMed, and the official release corroborate the publication. It is marked as an existing publication link with restricted automated full-text access; no numbers were extracted there. Some OpenReview pages required browser verification and were used only to confirm venue information. Core figures come from readable arXiv v3/PMLR text.
0.1 Candidate Pool and Random Selection
Candidates came from the official ICML 2025 PMLR 267 directory, matching the AI4Science rotation, a 12-month window, and no overlap with indexed topics from the previous 14 days. There were 28 candidates:
From Mechanistic Interpretability to Mechanistic Biology; 2. A
Variational Perspective on Generative Protein Fitness Optimization; 3.
Boosting Protein Graph Representations through Static-Dynamic Fusion; 4.
Elucidating the Design Space of Multimodal Protein Language
Models; 5. NMA-tune; 6. Aligning Protein Conformation Ensemble
Generation with Physical Feedback; 7. Diffusion on Language Model
Encodings for Protein Sequence Generation; 8. PPDiff; 9.
Flexibility-conditioned Protein Structure Design; 10. Protriever; 11.
All-atom Inverse Protein Folding; 12. CFP-Gen; 13. Protein Structure
Tokenization; 14. ReQFlow; 15. Bridging Protein Sequences and Microscopy
Images; 16. Beyond Atoms; 17. Pretraining GFlowNets for Molecular Graph
Generation; 18. Offline Model-based Optimization for Molecular
Discovery; 19. Foundation Molecular Grammar; 20. Kinetic Langevin
Diffusion for Crystalline Materials; 21. Open Materials Generation; 22.
All-atom Diffusion Transformers; 23. OmniArch; 24. LLM-SRBench; 25.
SCENT; 26. Retrieval Augmented Zero-Shot Enzyme Generation; 27.
Physics-Informed Weakly Supervised Interatomic Potentials; 28.
Physics-informed Temporal Alignment for PDE Foundation Models.
Running shuf -i 1-28 -n 1 returned
4. The paper has formal full text, over 14 quantitative tables/appendix tables, code, and weights; evidence was sufficient, with zero redraws. Candidate-pool entry: ICML 2025 PMLR 267.
1. In-depth analysis of the core paper
1.1 Title and Core Positioning
The innovation is not a larger protein model but a decomposition into four testable levels: supervision labels, generation space, network inductive bias, and training-data coverage. Tokenizer reconstruction error is only one limit. More critically, classification over topology-free codebook indices misaligns the objective with geometric similarity. The solution shifts from better tokenizers to new prediction units, continuous residuals, pairwise geometry, and teacher-smoothed supervision.
1.2 Motivation: Specific Problems and Quantitative Evidence
Multimodal PLMs jointly model amino-acid sequence () and 3D backbone (). An encoder maps structure to continuous latents, quantized into discrete tokens so a language model can process geometry as a vocabulary. This interface creates three measurable failures.
First, quantization loss. Table 1 (PDF p.3) gives continuous-latent reconstruction RMSD 1.3127 and TM-score 0.9733; quantization worsens them to 1.9806 and 0.9385. RMSD rises 0.6679, a 50.88% relative deterioration, while TM-score falls 0.0348. Discretization is not free compression.
Second, better tokenizer reconstruction does not imply better LM generation. Table 2 (p.3): ESM3's tokenizer RMSD 0.7248 greatly beats DPLM-2's 1.9806, yet with the same model architecture on CAMEO 2022, ESM3 codebooks yield generated RMSD 8.4424 versus 7.7025 for DPLM-2. Reconstruction error decreases by
63.41%, but generation error increases by
9.61%, directly challenging tokenizer reconstruction as the sole objective.
Third, index supervision is severely distorted. Table 3 (p.3): DPLM-2 CAMEO whole-index accuracy is 0.0864 versus bit accuracy 0.7720; PDB time-split values are 0.1188 versus 0.7932. Similar tokens may differ by one bit yet be treated as unrelated softmax classes; numerically nearby indices need not be geometrically close. This identifies more than generic insufficient capacity: a mismatch between supervision and structural metrics.
Why solve it? Unified models support unconditional sequence–structure generation, folding, inverse folding, motif scaffolding, and representations. Coarse structural symbols make that unification superficial. A well-designed 650M model approaching a specialized 3B folder would also lower research compute costs.
1.3 Narrative: Diagnosis to Combined Design
The argument has four turning points.
Separate the bottlenecks. Continuous/discrete reconstruction, tokenizer comparisons, and index/bit accuracy distinguish lossy tokenization from poor token prediction.
Change the supervision unit. Bit modeling replaces (2^K)-way classification with (K) binary predictions, exposing local differences. ResDiff models quantization residuals; Hybrid FM reinterprets encoder–PLM–decoder as a coordinate denoiser.
Acknowledge missing geometry in sequence Transformers. GeoDPLM adds residue-pair representations, pair biases, transitions, and optional triangle operations. REPA uses ESMFold as a teacher for smooth structural semantics.
Test whether modules are orthogonal. Table 7 shows more modules need not help: Geo+Bit diversity is 0.900, falling to 0.575 with FM. All* PDB RMSD 2.379 does not beat Bit+FM+ResDiff+SFT's 2.370. The authors recommend Geo + Bit rather than every module together.
Trade-offs are clear: discrete tokens retain scalable unified interfaces but lose continuous geometry; FM improves coordinates while ODE trajectories reduce diversity; SFT improves folding but sacrifices unconditional generation. AlphaFold-like triangle operations slow training 3.9×/6.8× without proportional gains.
Hybrid FM defines a coordinate denoiser (_=DecPLMEnc), with Euler step:
[ s(_t,t) +_t, ]
Each step first applies Kabsch alignment. REPA projects PLM hidden states through a three-layer MLP and maximizes cosine similarity to ESMFold representations:
[ {REPA}=-1L{i=1}^{L} (y_i,h_(h_i)). ]
1.4.2 Pipeline
Protein backbone x ──Enc──> Continuous z_cont ──LFQ──> K-bit z_quant ──> index
│ │ │
│ └─residual r──────────> ResDiff refinement
│ │
Sequence s ───────────────> Masked discrete-diffusion PLM <────────┘
│
┌───────────┼─────────────┐
│ │ │
Bit supervision Geo pair bias REPA←ESMFold
│ │ │
└───────────┴─────────────┘
↓
Dec backbone output / FM coordinate sampling
1.4.3 Mechanisms and Engineering Details
Mechanism
Design choice
Motivation
Key reported settings
Bit supervision
(K) binary classifiers replace (2^K)-way indices
Align label distance with local latent variation
Reuse LFQ bits directly; no separate K sensitivity study
Improve folding but lose eligibility for unconditional joint-generation evaluation
Multimer
PDB-Multimer + chain linker / position offset
Extend monomer modeling to interchain relations
Full dataset 11,614/291 train/validation; SFT filters to 3,462 examples, length 60–512 (Table 8, p.9)
1.5 Experimental Design and Quantitative Results
1.5.1 Data, Tasks, and Metrics
Data / task
Size or split
capability
Metric interpretation
CAMEO 2022 folding
Uses DPLM-2 evaluation; sample counts not restated
Temporally held-out folding
RMSD↓ measures coordinate error; TM-score↑ global topology
PDB date split folding
Date-separated; sample counts not restated
Generalization to recent structures between in- and out-of-distribution settings
RMSD↓, TM-score↑
Unconditional co-generation
40 samples per protein length (Figure 6)
Mode coverage in joint sequence–structure generation
FoldSeek clusters / samples = Diversity↑
CAMEO inverse folding
Structure-conditioned sequence generation
Sequence recovery and structural self-consistency
AAR↑, self-consistency TM-score↑
HumanPPI / DeepLoc
SaProt protocol
Structure-aware representations
Classification accuracy↑
PDB-Multimer
11,614 train / 291 validation; mean 2.88±1.66 chains
Multichain modeling
RMSD↓, TM-score↑
Main performance results have no repeated-run standard deviations or confidence intervals. PDB-Multimer chain/length deviations describe the dataset, not model uncertainty.
1.5.2 Complete Main-Results Matrix
Transcribed from Table 7 (PDF p.8): five metrics and ten configurations. Bold indicates the best comparable value per column. Dashes mean not evaluated; no values are invented.
Method
PDB RMSD↓
PDB TM↑
CAMEO RMSD↓
CAMEO TM↑
Diversity↑
DPLM-2 650M
5.307
0.8306
7.703
0.7936
0.700
Bit
3.221
0.9043
6.403
0.8380
0.825
Bit + FM
2.870
0.9099
6.183
0.8418
0.525
Bit + FM + ResDiff
2.788
0.9146
6.077
0.8456
0.525
Bit + FM + ResDiff + SFT
2.370
0.9270
5.847
0.8442
—
Geo + Bit (recommended)
2.551
0.9254
5.955
0.8520
0.900
Geo + Bit + FM
2.443
0.9261
6.172
0.8404
0.575
Geo + Bit + REPA
2.507
0.9264
6.192
0.8412
0.875
Geo + Bit + REPA + SFT
2.404
0.9322
5.754
0.8424
—
All*
2.379
0.9297
6.200
0.8398
—
Relative to DPLM-2 650M, Bit reduces PDB RMSD by 2.086, or
39.31%, and improves TM-score by 0.0737 (+8.87% relative). Recommended Geo+Bit reduces PDB RMSD by
51.93%, CAMEO RMSD by
22.70%, and increases diversity by 28.57%. Specialized SFT reaches PDB RMSD 2.370, a baseline-relative reduction of 55.34%.
The Strongest Evidence: Changing supervision while retaining 650M parameters lowers PDB RMSD 5.307→3.221; Geo+Bit reaches 2.551 with diversity 0.900. The main bottleneck is supervision geometry and structural bias rather than parameter count.
Table 4 (p.5) gives external CAMEO/PDB RMSD baselines: ESMFold 3B 3.9900/2.8400, ESM3 1.4B 6.3300/4.9003, MultiFlow 17.8400/15.6400. Bit+FM+ResDiff+SFT's PDB 2.3698 beats ESMFold by
16.56%, but its CAMEO 5.8472 is worse by
46.55%. Matching or beating specialized folders applies only to particular splits/metrics, not universally.
1.5.3 Incremental Ablations and Behavioral Shifts
Design choices move the Pareto trade-off between folding RMSD and generative diversity
Bit supervision produces the largest accuracy jump; FM further lowers RMSD while sharply reducing unconditional diversity.
PDB RMSD (lower is better)
Source: Table 7 (PDF p.8). One RMSD y-axis only; exact diversity is available via hover/focus tooltips and the adjacent table, without dual axes.
Geometry ablation (Table 5, p.6): without SFT, Base scores 5.521/7.703; pair bias/transition gives 4.823/7.244, adding structure transition gives 3.883/6.550. Triangle updates regress to 4.837/7.197 and triangle attention to 4.415/6.973. These complex modules slow training 3.9× and 6.8×(p.6) without beating inexpensive transitions.
Highlighted Behavioral Shifts. Index→bit produces a large performance jump: PDB index accuracy 0.1188→0.2641, bit accuracy 0.7932→0.8648, and RMSD 5.3071→3.2213 (Table 3, p.3). Bit→Bit+FM changes the objective trade-off: RMSD improves another 10.90%, but diversity falls 0.825→0.525, or
36.36%. This is not smooth improvement with added modules; moving from stochastic discrete denoising to more deterministic ODE trajectories abruptly changes mode coverage.
1.5.4 REPA, Inverse Folding, and Representations
Table 6 (p.7): REPA reduces DPLM-2 PDB RMSD 5.521→4.919 (−10.90%) and GeoDPLM 4.823→4.340 (−10.01%), helping both architectures. Table 13 (p.16): CAMEO inverse-folding AAR rises from 0.4962 for DPLM-2 650M to 0.5586 for Bitwise and 0.5681 for Geo+Bit+REPA, +0.0719 absolute (relative
14.49%). Table 14 (p.17): Bitwise reaches HumanPPI 88.89%, beating SaProt 86.41% and DPLM-2 84.44%, but DeepLoc 83.39% trails SaProt 85.57%. Transfer exists but is not universal.
1.5.5 Multichain Data
Table 9 (p.8): increasing monomer tokenizer data from 200K by 1.2M reduces PDB-Multimer reconstruction RMSD 9.973→6.873 (−31.08%) and raises TM-score 0.694→0.784, showing local geometric transfer. Table 10 (p.9): positional offsets reduce tokenizer RMSD 6.873→5.886 (−14.36%), but ESMFold's folding TM-score 0.850 still beats DPLM-2/concatenation/offset at 0.768/0.771/0.767. Table 11: multimer-only SFT worsens CAMEO RMSD to 10.989; restoring SwissProt recovers 6.410. Multichain data should supplement, not replace, monomer data.
1.5.6 Hyperparameter Sensitivity and Statistical Significance
No systematic sweeps cover learning rate, K, codebook size, FM steps, REPA weight, or positional offsets; no multi-seed mean±SD is given. Confirmed details are pair/structure dimensions, attention heads, three ESMFold recycles, speed logged every 10K steps, and roughly 10× fewer FM sampling steps.
Thus 5.307→3.221 is a large effect under the paper's setting, not a demonstrated statistically significant result. Whether bit modeling plateaus beyond some K is unknown. Table 7 shows a behavioral trade-off, not a phase transition established through a continuous sweep.
1.6 Value Analysis
The Substantive Contribution is a diagnose–design–combine framework. Tokenizer reconstruction alone is insufficient; jointly inspect predictability, coordinates, topology, diversity, and transfer. This framework may outlast individual modules.
Three reusable designs follow. LFQ/VQ models can test bitwise or hierarchical labels instead of false equidistance among indices. REPA can align general generators with specialized scientific hidden representations rather than only final predictions. Combined ablations must report orthogonality: Geo, Bit, FM, and REPA helping separately does not imply they help together.
Potential transfers include RNA, small-molecule conformations, crystals, and protein–ligand complexes: discrete modalities coexist with continuous symmetric geometry. Bits suit the discrete side, equivariant/pairwise modules the geometric side, and physical energy the constraint side. Geo+Bit is particularly economical, retaining folding and diversity without FM sampling or teacher precomputation.
1.7 Limitations
1.7.1 Author-Acknowledged Limitations
Appendix B, Discussions and Limitations (PDF p.14), acknowledges four points:
Discrete representations remain lossy. Bits only mitigate quantization and cannot capture full continuous geometry. Residue/backbone modeling omits all-atom interactions.
No physical constraints.
Without explicit energy priors, improved structural metrics do not establish physical realizability.
Scaling remains unverified. Analysis extends only to 3B; effectiveness at larger scales is unknown.
Scarce high-quality multichain data. Length-filtered PDB-Multimer SFT has only 3,462 examples, limiting family-external generalization.
1.7.2 Additional Criticisms
[Argument gap] Claims of beating specialized folders depend on the split.
Table 4's SFT model beats ESMFold on PDB RMSD 2.3698 vs 2.8400 (16.56%), but loses on CAMEO 5.8472 vs 3.9900 (46.55% worse). Matching/outperforming claims need dataset qualifications.
[Metric validity] RMSD and TM-score do not establish biological function or synthesizability.
All main results measure structural similarity. The paper provides no wet-lab validation, stability after energy relaxation, Rosetta energy, pLDDT/calibration, or functional assays. An RMSD of 2.370 therefore does not establish that a generated protein performs the intended function.
[Statistical significance] No multiple-seed results. Tables 4–7 and 13–14 provide neither performance standard deviations nor confidence intervals. Gains such as REPA improving GeoDPLM's CAMEO TM-score from 0.8128 to 0.8217, a difference of 0.0089, may be affected by training variability. These are observed improvements, not demonstrated statistically significant gains.
[Experimental design] Baseline values differ across tables. DPLM-2's PDB RMSD is 5.3071/5.307 in Tables 4/7 but 5.521 in Tables 5/6; the abstract states 5.52→2.36. Different training or evaluation configurations are insufficiently distinguished in the overall narrative, making the relative improvement range from 55.34% to 57.07%.
[Module interactions] All* is not optimal, weakening the case for stacking everything.
All* achieves PDB RMSD 2.379, worse than the SFT route's 2.370, and CAMEO 6.200, worse than 5.754. Modules therefore interact negatively: the design space is not an additive collection of components. The authors report this candidly but do not provide controlled second-order experiments explaining the interactions.
[Underperforming subtask] FM trades diversity for accuracy. Bit→Bit+FM improves PDB RMSD from 3.221 to 2.870, but diversity falls from 0.825 to 0.525 (−36.36%). FM may therefore be the wrong choice when diverse backbone discovery is the objective. The paper attributes this to ODE properties but does not test that causal explanation against a stochastic sampler.
[Missing fair comparisons with key competitors] Model sizes and task specialization are asymmetric. ESMFold 3B is a dedicated folding model, ESM3 1.4B is a multimodal model trained on different data, and the core configuration includes folding SFT. Parameter efficiency cannot therefore be separated from task supervision and training data. A head-to-head comparison needs matched training tokens, structure data, and SFT.
[Reproducibility risk] The public release is incomplete.
The official repository explicitly provides DPLM-2 Bit weights, but its README still says “full implementation will be released soon.” The verifiable checkpoint is the 2.62GB 650M Bit model; equally clear end-to-end reproduction instructions for the full GeoDPLM, ResDiff, FM, and REPA combinations are unavailable. This makes the combined results in Table 7 difficult to verify end to end.
2.
Related-work retrospective: the chain of problems and technical origins
2.1 DPLM — 2024.07, ICML 2024
Problem solved. DPLM introduces absorbing discrete diffusion into protein sequence language modeling, bringing unconditional generation, representation learning, motif scaffolding, and conditional inverse folding into one model. It addresses the unnatural autoregressive ordering of protein sequences and inflexible condition composition, providing the sequence-pretrained foundation for this work.
Remaining gap. DPLM natively handles only discrete amino-acid sequences; structure can be an external condition but cannot be jointly generated. Plausible sequences do not guarantee plausible 3D backbones, leaving unified protein models without a structural channel.
How the core paper responds. DPLM-2.1 retains discrete diffusion and its scalability, discretizes structure for joint training, and then addresses the supervision and geometry problems introduced by discrete structure.
Inherited design. Denoising language modeling at arbitrary mask ratios carries over from DPLM to DPLM-2/2.1. The core change concerns the granularity of structural targets, not the diffusion framework.
2.2 DPLM-2 — October 2024 preprint / ICLR 2025
Problem solved. DPLM-2 compresses 3D backbones into structural tokens using an LFQ tokenizer and models them jointly with sequence tokens. This supports sequence–structure co-generation, folding, inverse folding, and motif scaffolding. It is the direct baseline and experimental platform for the focal paper.
Remaining gap. The unified interface is established, but tokenizer quantization loses detail, index prediction accuracy is low, and the sequence Transformer lacks geometric bias. The CAMEO index accuracy of 0.0864 in Table 3 directly measures this gap.
How the core paper responds. DPLM-2.1 reframes “whether to discretize” as “how to discretize, supervise, and recover continuous information,” proposing Bit, ResDiff, FM, Geo, and REPA.
Inherited design. LFQ's bitwise quantization was originally an internal tokenizer implementation detail. Here it becomes an explicit language-model supervision unit—the key transition from a hidden design choice to a learning objective.
2.3 ESM3 — July 2024 preprint / January 2025 Science
Problem solved. ESM3 encodes sequence, structure, and function as multiple tracks of discrete tokens and scales to 98B parameters. Its central contribution is demonstrating that multimodal protein language models can accept arbitrary modality combinations as prompts and generate experimentally validated, evolutionarily distant fluorescent proteins.
Remaining gap. Scale alone does not establish whether a structural tokenizer suits language modeling. The controlled comparison in Table 2 shows that the ESM3 tokenizer reconstructs well at RMSD 0.7248, yet produces generation RMSD 8.4424 when inserted into the same DPLM-2 architecture, worse than the DPLM-2 tokenizer's 7.7025.
How the core paper responds. The paper uses ESM3 to show that reconstruction quality is an insufficient proxy, and instead optimizes predictability and geometric bias. Larger codebooks and lower reconstruction error are therefore not its only priorities.
Inherited design. The work inherits ESM3's unified multitrack discrete interface, adding bit-level correlations and continuous repair to retain that language interface while reducing geometric costs.
2.4 REPA — October 2024 preprint / ICLR 2025 Oral
Problem solved. REPA aligns noisy-state hidden representations in image diffusion Transformers with pretrained visual encoders, reporting 17.5× faster training and FID 1.42. It recasts difficulty in generation as difficulty in learning high-quality representations.
Remaining gap. Original REPA targets images, 2D patches, and DINO-like teachers. Its transfer to discrete protein structures, and the appropriate scientific teacher, remained open questions.
How the core paper responds. DPLM-2.1 transfers REPA to residue-level structural representations using the ESMFold folding trunk as teacher. Table 6 shows benefits for both the pure LM and GeoDPLM.
Inherited design.
The transfer evolves from a self-supervised visual teacher guiding an image generator to a specialized folding model guiding a general protein generator. A learnable multilayer ensemble removes the need to choose alignment layers manually.
3. Technical evolution
2024 DPLM [discrete diffusion; sequence only; unified generation/representation]
│
├──> 2024/2025 DPLM-2 [LFQ structure tokens; joint sequence–structure distribution]
│ │
│ ├── Gap A: lost quantization residuals ──> ResDiff / Hybrid FM
│ ├── Gap B: index labels lack geometry ─> Bit supervision
│ └── Gap C: Transformer lacks 3D bias ─> GeoDPLM
│
2024 ESM3 [sequence/structure/function tracks; massive scale]
└── Lesson: a unified discrete interface scales, but tokenizer reconstruction ≠ generation quality
2024 REPA [external representations smooth generative supervision]
└──> 2025 DPLM-2.1 [alignment with an ESMFold teacher]
2025 DPLM-2.1 convergence
[Bit + Geo as the recommended backbone; choose FM/ResDiff/SFT by objective; Multimer expands data coverage]
The underlying driver is not simply model scaling, but a shift from a unified interface to consistent metrics: early work asks whether sequences and structures can both become tokens. Later, token predictability, geometric proximity, and coordinate accuracy prove misaligned, prompting the return of continuous models, equivariant biases, and specialized teachers. The turning point is acknowledging that linguistic unification does not imply physical unification. Community attention also shifts from folding accuracy alone to the three objectives of accuracy, diversity, and efficiency, as reflected in Table 7 and the training-speed analysis.
4. Cross-paper comparison
Work
Publication date
Conference/publication
Core contribution
Representative evidence
Relationship to the core paper
DPLM
2024.07
ICML 2024
Discrete diffusion LM for protein sequences
Unified generation, representation, and conditional control
5. Open challenges and actionable research opportunities
5.1 Theory
Bounds on label–metric consistency. Fix the LFQ encoder, codebook, and parameter count; compare index CE, bit BCE, Gray code, hierarchical softmax, and latent-distance-based soft labels. Under identical training-token budgets, determine the tightest upper bound on structural RMSD implied by token classification error.
Identifiability of joint discrete–continuous representations. Compare pure tokens, tokens + residuals, and direct SE(3) flow on identical data, controlling compute and supervision. Test whether residuals only repair local bond lengths or can change global topology.
Causal explanations for non-orthogonal modules. Run a full (2^4) factorial experiment over Bit/Geo/FM/REPA with at least 3 seeds and fit second-order interaction terms. Test whether REPA and Bit actually saturate each other's benefits because they duplicate supervision smoothing.
5.2 Engineering and Applications
Fair folding comparisons under equal budgets. Match PDB/SwissProt data, 650M parameters, training FLOPs, folding SFT, and sampling time across DPLM-2.1, ESMFold-small, ESM3-open, and Proteina. Report CAMEO/PDB results, inference seconds, and peak memory together.
The accuracy–diversity Pareto frontier. Add stochastic interpolants, temperature, or noise injection to Bit+FM and compare directly with the pure ODE. Fix 40 samples/length and plot RMSD against FoldSeek diversity on a single Pareto chart to test whether the 36.36% diversity loss can be recovered.
Additional physical-feasibility evaluation. Evaluate clash rate, Ramachandran outliers, Rosetta/Amber relaxation, and short MD stability on the same generated structures. Correlate these with RMSD/TM-score to test whether structural metrics proxy actual physical quality.
The reproduction gap. Once all Geo/ResDiff/FM/REPA configurations are public, reproduce Table 7 using default repository commands. Record wall-clock time, GPU-hours, and the cost of precomputing ESMFold representations to estimate the compute cost per 1% RMSD improvement.
5.3 Emerging Directions
All-atom and ligand conditioning. Extend the 4 backbone atoms to side chains and ligands, comparing residue tokens + atom residuals with all-atom flow. Evaluate protein–ligand poses, interface RMSD, and binding energy instead of single-chain TM-score alone.
Long-tail generalization across multiple chains. Stratify by chain count, interface area, and homologous family; compare monomer-only, multimer-only, and mixed curricula. In particular, revisit the source of negative transfer behind the reported multimer-only CAMEO RMSD of 10.989.
An experimental feedback loop. Select equal numbers of candidates from Geo+Bit and Bit+FM generations and blindly evaluate expression rate, thermal stability, and function. Use experimental success rates to test whether diversity 0.900 predicts discovery value better than RMSD 2.370.
Uncertainty and abstention. Beyond CAMEO/PDB, construct low-homology, very long, and disordered-protein sets. Use ensemble/conformal methods to provide folding-error intervals and compare them with point TM-score predictions, establishing thresholds for when generated structures should not be trusted.
6. Other recent work worth watching
Protein Structure Tokenization: Benchmarking and New
Recipe (2025.07, ICML 2025)
Systematically evaluates tokenizers through local substructures, efficiency, and codebook utilization. AminoAseed improves over ESM3 by an average 6.31% across 24 supervised tasks, making it a strong candidate for DPLM-2.1's next tokenizer head-to-head comparison. The paper | Code/weights
Retrieval Augmented Zero-Shot Enzyme Generation for Specified
Substrate (2025.07, ICML 2025)
Combines substrate-similarity retrieval with conditional discrete diffusion to define zero-shot substrate-specific enzyme generation. Its connection from general sequence generation to evaluable catalytic objectives merits attention.
The paper | Code not found
Foundation Molecular Grammar (2025.07, ICML
2025)
Uses multimodal foundation models to induce interpretable molecular graph grammars connecting images, text, and molecular graphs. Its route of obtaining structured symbols before generation parallels the metric problem of protein structure tokens. The paper | Code
Open Materials Generation with Stochastic
Interpolants (2025.07, ICML 2025)
Handles periodic lattices, coordinates, and atom types within a unified stochastic-interpolation framework, providing an important reference for transferring joint continuous/discrete generation to crystals.
The paper |
Code
All-atom Diffusion Transformers (2025.07, ICML
2025)
Uses a shared latent space to generate both periodic materials and nonperiodic molecules, scaling to 500 million parameters. In contrast to DPLM-2.1, it reduces domain-specific inductive biases in exchange for unification and speed. The paper | Code
LLM-SRBench (2025.07, ICML 2025)
Across 239 symbolic-regression problems in four scientific domains, the best system achieves only 31.5% symbolic accuracy. This underscores the need for AI4Science evaluations to guard against memorization and distinguish discovery from reproducing formulas. The paper | Code link on the conference page
Bridging Protein Sequences and Microscopy Images with Unified
Diffusion Models (2025.07, ICML 2025)
CELL-Diff generates bidirectionally between protein sequences and fluorescence microscopy images, extending protein multimodality from sequence–structure to sequence–cellular-localization phenotype.
The paper | Code not found
7. Conclusion
DPLM-2.1's most important conclusion is not a single SOTA number, but an engineering principle broadly relevant to multimodal scientific models: A symbolic interface must preserve neighborhood relationships in the underlying continuous space; otherwise, scalable language modeling optimizes the wrong metric.
The 650M model reduces PDB RMSD from 5.307 to 3.221 through Bit, then achieves a more balanced point with Geo+Bit at RMSD 2.551 and diversity 0.900, demonstrating that design can substitute for some parameter scaling.
The limitations remain: it still trails ESMFold substantially on CAMEO; main results lack multiple seeds; no physical-energy or experimental validation is provided; the full combination is not optimal; and complete component implementations are not equally public. The most reliable next step is therefore equal-budget Pareto evaluation across splits with physical and experimental endpoints, rather than continued module stacking.
Sources and verification notes
All core quantitative data come from arXiv 2504.11454v3
PDF (2025-06-12): Tables 1–3 (page 3), Table 4 (page 5), Table 5 (page 6), Table 6 (page 7), Tables 7–9 (page 8), Tables 10–11 (page 9), limitations (page 14), Table 13 (page 16), and Table 14 (page 17). Page numbers refer to the PDF's displayed pages.
Conference and publication information was verified through PMLR
; code, weights, and data were checked by visiting the official project page, GitHub, and Hugging Face.
Relative improvements were recalculated from the original table values and rounded to two decimal places; unreported items remain “—”.
Limits of this report: no model training or protein-structure evaluation was run. Missing paper details cannot establish seed variability, total training FLOPs, complete hyperparameter curves, or wet-lab success rates.
The HTML passed Warm Editorial structure and offline-dependency validation (0 warnings). Although the environment contained the Playwright package, the specified
/usr/bin/chromium with /opt/pw-browsers
were absent, and no other Chromium executable was found. Following the instruction not to run playwright install, this report could not include real-browser screenshot comparisons for light, dark, and mobile layouts. Responsive, theme, and print rules therefore received static checks only; visual-rendering acceptance was not claimed.
Index status: local ml-report-index.md
has been updated; persistent synchronization occurs at delivery.