RnG: KV-Cache as an Implicit 3D Representation

0. Report Metadata and Transparency Statement

✅ = fetched and confirmed reachable during the original review; ⚠️ = credible source, but not independently opened.

2.1 Positioning in One Sentence

The key claim is not that caching is fast, but that the network's intermediate activations already encode geometry on which it was never supervised. RnG's engineering makes that hypothesis queryable, renderable, and exportable.

2.2 Motivation and Problem Evidence

The Precise Problem

Feed-forward generalizable reconstruction, from DUSt3R to VGGT, reduced sparse-image-to-3D inference to subsecond speed within two years. Yet its training objective asks only for visible geometry.

How the Problem Is Quantified

Rather than an abstract incompleteness rate, the paper uses Chamfer Distance (CD) as its decisive metric. In Table 2, VGGT has strong source-view depth Rel of 5.960, but complete-3D CD is 0.0260. RnG reaches 0.0067, 74.2% lower (3.88×). Accurate visible geometry alone does not yield complete geometry.

The reverse comparison is more compelling: Matrix3D, then the only unified model generating unseen regions, has pose-free CD 0.0670— 2.58× worse than VGGT's 0.0260 without generation. Before RnG, the field faced an either-or trade-off:

Why It Matters

• Applications — physical simulation, collision detection, 3D printing, and content creation require closed geometry; front-facing point clouds are insufficient.
• Systems — unified diffusion inference at 27s prevents interactive applications.
• Representations(the most interesting point)—if reconstruction latents already contain complete geometry, generation needs no separate generative model, only a different readout.

Recent approaches add geometry to generative diffusion models, including Zero-1-to-3, Matrix3D, and Gen3R. RnG instead opens a generative output path from a reconstruction model.

2.3 The Paper's Argument

The argument follows a chain with clear turning points, rather than merely proposing a successful module:

Step 1: State a Falsifiable Hypothesis

This is the paper's pivot: if true, a decoder should recover complete geometry without retraining a generative model.

Step 2: Add a Readout Channel Without Rewriting the Representation

To test whether complete 3D already exists in the latent space, target-view queries must not change reconstruction results. Otherwise existing geometry cannot be distinguished from information introduced through target views. This directly motivates the causal mask.

Step 3: Extract Engineering Benefits from the Control

Once source K/V is independent of target views, it can be cached. The implicit 3D representation becomes cached K/V tensors, and each new view is a cross-attention query. Table 4: FLOPs fall from 12.26T to 2.29T (5.35×), with 85ms per view, faster than Matrix3D's 27s by 317.6×.

Step 4: Define the Contribution Through What Is Absent

RnG adds no diffusion, GAN, 3DGS, NeRF, or new 3D representation—only a binary mask and two output heads.

Implicit Design Trade-Off

Although unstated, Tables 2/3 clearly show: texture detail is traded for geometric completeness and speed. RnG's PSNR 26.276 trails LVSM-100K's 27.522 (−1.246 dB), and LPIPS 0.0975 trails 0.0895. In return, CD becomes available at 0.0067 and inference accelerates 317×.

2.4 Methodology and Mechanisms

2.4.1 Overall Pipeline

2.4.2 Core Mechanism: Reconstruction-Guided Causal Attention

Let {s} denote source-token indices and {t} denote target-token indices:

Semantics: Source-view queries are forbidden from accessing target-view keys. This is not the standard autoregressive triangular mask, but a task-level one-way barrier that insulates reconstruction from generation. The paper calls it reconstruction-guided.

• Byproduct 1 · Shared Parameters: Source and target paths use identical weights, with no extra parameters.
• Byproduct 2 · Target-View Invariance: Whether querying one or 1,000 target views, source reconstruction is bitwise identical—an architectural guarantee rather than a learned approximation.

2.4.3 KV-Cache as an Implicit 3D Representation

Stage 1 · Reconstruction and Caching(target-independent, performed once offline):

Cache K/V for every source token at every layer. These tensors are the scene's 3D representation.

Stage 2 · Generation and Querying(the only step repeated for each new view):

Linearly projected target-view Plücker rays produce Q_t and directly query the cache, entirely bypassing source-branch forward computation.

2.4.4 Key Mechanisms

2.4.5 Training Objective

In L_pmap, the second term provides gradient-domain supervision for normals and sharp boundaries; the third term −α log Σ_t regularizes uncertainty, preventing the network from driving Σ to zero everywhere to evade supervision.

2.4.6 Engineering Details and Hyperparameters

3.1 Benchmarks and Protocol

3.2 Main Results Matrix (Table 2, GSO, Four Input Views)

Original Table 2 caption: "Quantitative comparison. We evaluate the reconstruction and generation ability of all models on the GSO dataset. '—' means that the model is not capable of delivering that result."

※ MUSt3R's identical RA@5 and RT@5 may reflect typesetting or HTML extraction errors; the PDF could not be checked.

3.3 Relative Improvements

3.4 Ablation Study (Table 3)

Incremental Decomposition

Full attention wins eight of ten metrics. The title and abstract present reconstruction-guided causal attention as a primary contribution, yet its value lies entirely outside quality, in two properties: (1) architectural target-view invariance; (2)enables KV caching and a 5.35× FLOP reduction. Both are important, but the narrative can imply that masking improves quality.

A candid formulation would be: “Small regressions of 0.01–1.07 on eight of ten metrics buy 5.35× computational savings and a cacheable 3D representation.” An excellent trade-off, but still a trade-off.

3.5 Efficiency (Table 4)

caption: "Efficiency comparison of models with and without KV-Cache for inferring a single novel-view appearance and geometry."

3.6 Sensitivity to Input-View Count (Table 5)

Decay pattern: Gain ratio = 2.873/3.951 = 0.727, 1.332/2.873 = 0.464, showing geometric decay. Extrapolating the later ratio r≈0.464 gives remaining gain ≈1.332×r/(1−r)≈1.15dB and an asymptote around 28.7–28.8dB.

3.7 Statistical Significance

The Substantive Contribution

It expands the candidate space of 3D representations rather than merely adding another unified model. Over five years, representations progressed through meshes, voxels, NeRF, 3DGS, triplane latents, and structured latents—all data structures designed specifically for 3D. RnG claims that intermediate activations of a general Transformer can fill this role if invariant to queries, thereby reducing representation design to attention-mask design.

Independently Reusable Designs

Implications for Future Work

The resulting research space follows cache semantics:

• Can a KV-cache 3D representation be edited? Perform object removal, replacement, or material edits directly on cached tensors without recomputing the encoder.
• Can it be compressed or quantized? Reducing 400MB to 40MB would create a neural compression format for 3D assets.
• Can caches be concatenated? Does joining two scene caches merge scenes? This directly concerns scene-level expansion.
• Can it be indexed by time? Adding time turns the KV-cache into explicit world-model memory.

This last question connects most closely to generative/world models: long-term video consistency depends on the form of memory, and RnG offers an answer— the reconstruction network's K/V tensors.

Category I: Author-Acknowledged (§5, Quoted)

(1) Missing detail

Evidence: LPIPS 0.0975 vs LVSM 0.0895 (8.9% gap), PSNR −1.246dB.

(2) World-origin definition

(3) Multi-view 3D accumulation

A single query produces a pixel-aligned point map(2.5D); closed 3D still requires multiple queries and fusion. Complete geometry is assembled after querying rather than produced in one forward pass.

Category II: Eight Independent Criticisms

Criticism 1 · Gaps in the argument — The headline mechanism adds no quality benefit, but is not positioned accordingly

In Table 3, Ours-15K-FullAttn wins or ties eight of ten metrics against Ours-15K(LPIPS 0.1193 vs 0.1243, a 0.0050 gap, compared with the 0.0268 improvement from 15K→40K training: 18.7 %). The abstract emphasizes causal attention as a performance contribution, whereas evidence supports preserving performance while enabling caching.

Criticism 2 · Missing fair baseline comparisons — Equal 40K steps do not mean equal compute

The paper compares against LVSM-40K(24.619) rather than LVSM-100K(27.522) to claim better data efficiency (+1.657dB). But RnG inherits VGGT's large-scale, multidataset pretraining. Its own Ours-15K-scratch quantifies this prior: PSNR 20.784 (−4.072 dB), RA@5 8.252 (−73.398 pp). Including pretraining compute undermines the equal-step comparison.

Missing controls: Initialize LVSM from VGGT too, or fine-tune VGGT on the same Objaverse data for 40K steps using its original objective before attaching LVSM. These separate architectural benefits from pretrained priors, and hidden-geometry readout from easier-domain adaptation.

Criticism 3 · Metric validity — CD alone supports completeness, without a transparent or decomposed protocol

• The paper does not specify directional versus bidirectional CD, scale alignment, or sampled point count.
• An accuracy / completeness decomposition is also missing. CD combines two terms; filling an object solid can improve completeness without accuracy while retaining an attractive total.
• VGGT's CD=0.0260 is itself ambiguous: it cannot generate unseen surfaces, so how is its complete point cloud constructed? Prediction→ground-truth CD favors visible-only predictions, whereas bidirectional CD penalizes missing backsides. The 3.88× gap means very different things under these protocols.

Criticism 4 · Experimental design issues — The highest-weight loss contributes nothing

Ours-15K-w/o cam with Ours-15K: PSNR 24.850 vs 24.856 (−0.006), LPIPS 0.1235 vs 0.1243 (better), source-depth Rel 0.739 vs 0.742 (better). Thus the pose loss λ_c=1, the largest weight, contributes zero or slightly negatively to appearance and geometry. A reasonable inference is that VGGT pretraining already supplies pose ability, with L_cam maintaining rather than learning.

The paper neither discusses this nor sweeps λ_c. If reducing λ_c to 0.1 or 0.01 preserves pose accuracy, more effective gradient budget could go to appearance, partly addressing missing detail.

Criticism 5 · Uneven evaluation coverage — No quantitative real-scene results

All numerical conclusions rely on GSO: 1,030 objects, clean backgrounds, orbiting cameras, synthetic renders with similar Objaverse-LVIS training data. CO3D has one qualitative figure, and the authors acknowledge potential ground-truth misalignment, yet claim real-time interactive applicability. An 85ms/view model validated only on synthetic objects lacks evidence for practical real-time interaction. No numbers cover indoor/street scenes, handheld trajectories, severe occlusion, or non-Lambertian materials.

Criticism 6 · Gaps in the argument — World-origin ambiguity is methodological, not merely preprocessing

The paper suggests improving preprocessing, but the architecture ĝ_{s=1} = [I₃ₓ₃ | (0,0,−1)ᵀ] takes the first source view and fixes it as the world frame, implicitly defining the object center through the intersection of source optical axes Implicit. Training cameras all look at the world center, making this definition work. The model therefore learns the dataset acquisition protocol rather than a geometric invariant.

Criticism 7 · Missing scalability analysis — 5.35× fewer FLOPs yield only 2.51× speedup

The 2.13× discrepancy in Table 4 points to a memory-bandwidth bottleneck. At 256×256 with patch size 8, each view has 1024 tokens; four sources across 24 layers require hundreds of MB of K/V. Proposed extensions imply:

The cache's main weakness is scaling with observation count; a good 3D representation should scale with scene complexity independently of observations. A 3DGS scene does not grow 100× because 100 additional photos were taken. Cache compression, pruning, and deduplication are not discussed.

Criticism 8 · Gaps in the argument — Signs of a capacity ceiling remain unchallenged

Table 5's gain ratios 0.727→0.464 suggest geometric decay toward approximately 28.7 dB. If true, even 16 or 32 views may not substantially beat LVSM-100K's 27.522. The bottleneck may be D_RGB capacity, 256×256 resolution, or lack of generative high-frequency priors, consistent with acknowledged missing detail. The paper stops at eight views; one 16-view point could distinguish bounded geometric decay from unbounded logarithmic growth.

Limitations of My Analysis

• GitHub and Hugging Face could not be opened independently, so actual code/weight release could not be verified, including training code and licensing. Evaluation code would be the best way to resolve criticism 3's CD questions.
• I did not read the PDF; table data came from structured arXiv HTML extraction, so MUSt3R's identical RA@5/RT@5 may be an extraction artifact.
• Only textual descriptions, not Figures 13–15 themselves, were available for supplementary failure cases; analysis relies on the authors' account.
• No independent reproduction or numerical verification was performed; performance figures are cited from Tables 2/3/4/5.

6.1 DUSt3R — 2023.12 (arXiv)/ CVPR 2024

Problem addressed

Conventional MVS/SfM needs known or estimated camera parameters; pose failure breaks the pipeline. DUSt3R reformulates paired-image reconstruction as point-map regression, directly predicting each pixel's 3D coordinates in a common frame while relaxing strict projective-camera constraints. In one sentence: one regression task replaces an entire geometry pipeline.

Unresolved gaps

(a)Inherently pairwise —N images require O(N²) forward passes plus expensive global alignment, compromising a pure feed-forward approach; (b) scale ambiguity; (c)visible regions only —pixel alignment means no pixel, no geometry: an intrinsic representational limitation.

How the Core Paper Responds

RnG inherits point maps as its geometry output (D_pmap head and L_pmap loss), but cleverly bypasses rather than solves gap (c): if point maps need pixels, let query views supply them. Plücker rays also cover unseen regions, where the D_pmap output becomes inferred geometry.

Transfer of Key Designs

Point maps and heteroscedastic uncertainty-weighted losses pass almost unchanged from DUSt3R through VGGT to RnG—the chain's most stable technical inheritance.

6.2 LRM — 2023.11 (arXiv)/ ICLR 2024

Problem addressed

Before LRM, single-image 3D relied on per-scene SDS optimization lasting tens of minutes. LRM demonstrates that large-scale feed-forward regression can replace optimization: a large Transformer trained on Objaverse produces a triplane NeRF in five seconds. In one sentence: 3D reconstruction becomes a large-model problem rather than an optimization problem.

Unresolved gaps

(a) With a single image, reconstruction and generation are conflated : the backside must be generated, but LRM cannot distinguish observed certainty from unseen guesses. (b) Triplane NeRF limits geometry accuracy. (c) Cameras must be normalized.

How the Core Paper Responds

Gap (a) is RnG's direct target. The causal mask separates source reconstruction, proven target-independent, from target generation. Distinguishing observed from guessed information moves from no mechanism in LRM to an architectural guarantee in RnG.

Transfer of Key Designs

The large feed-forward model/Objaverse paradigm and zero-shot GSO evaluation descend from LRM. RnG's 113.5K Objaverse-LVIS training and 1,030-object GSO evaluation continue this line.

6.3 LVSM — 2024.10 (arXiv)/ ICLR 2025 Oral

Problem addressed

Even after LRM, novel-view synthesis used 3D biases such as NeRF/3DGS, epipolar constraints, and plane sweeping. LVSM removes them all, retaining a pure Transformer from image tokens and target Plücker rays to pixels. It improves over prior SOTA by +1.5 ~ 3.5 dB PSNR and beats earlier methods with only one or two GPUs. In one sentence: data and scale can replace explicit 3D inductive biases.

LVSM Has Two Variants, Essential for Understanding RnG: Encoder-Decoder Compress inputs into a fixed number of 1D latent scene tokens—a learned implicit 3D representation; Decoder-Only remove the intermediate representation entirely for higher quality at higher cost.

Unresolved gaps

(a)Ground-truth camera poses are required , forcing RnG to supply VGGT-estimated poses for the VGGT+LVSM baseline, whose PSNR is only 19.394, 8.1dB below LVSM's own 27.522, demonstrating severe error propagation; (b) no geometry output; (c) encoder–decoder latents are implicitly 3D but learned specifically for rendering rather than aligned with reconstruction, so geometry cannot be read out.

How the Core Paper Responds

• For (a): VGGT makes pose an endogenous output instead of an input. RnG's RA@5=85.146 exceeds VGGT's 74.239 by 10.9pp, suggesting generation actually improves pose estimation.
• For (b): parallel D_pmap heads generate RGB and geometry from the same tokens.
• For (c): This Is the Deepest Response. LVSM learns new latents; RnG reuses reconstruction K/V, naturally aligned with geometry because that was its training objective. Reusing an existing representation instead of learning a new one is the chain's largest conceptual shift.

Transfer of Key Designs

The transfer is direct: Plücker-ray maps as target-view queries are structurally identical in RnG and LVSM. RnG can be understood as replacing LVSM's encoder with VGGT and its latent tokens with a KV-cache. RnG also acknowledges sharing LVSM's missing-texture-detail problem.

6.4 VGGT — 2025.03 (arXiv)/ CVPR 2025 Best Paper

Problem addressed

VGGT addresses DUSt3R's pairwise limitation and alignment cost with alternating frame/global attention, using any number of views to simultaneously predict cameras, depth maps, point maps, and 3D point tracks, < 1 second, outperforming optimized methods without postprocessing. In one sentence: one network handles all 3D perception tasks in one forward pass.

Unresolved gaps

One Large Gap: It is purely discriminative: outputs are functions of observations rather than a model of the scene.(a) Visible-only geometry, with no backside information in CD=0.0260; (b) no generation; (c) RnG's insight: the latent space may already know the answer but lack an output channel.

How the Core Paper Responds

RnG responds with minimal intervention: retain the architecture and pretrained initialization, adding only a binary attention mask, two output heads, and target rays, then fine-tune for 40K steps. All VGGT capabilities are preserved and improved(RA@5 74.239→85.146; source-depth Rel 5.960→0.584), while generation is added.

Transfer of Key Designs

What transfers is the weights themselves, the chain's only literal inheritance rather than conceptual borrowing. Thus the collapse of Ours-15K-scratch (PSNR −4.072dB, RA@5 −73.4pp) is the most important ablation, quantifying that inheritance. RnG discovers an unused outlet in VGGT rather than building a wholly new model. This is both its elegance and the source of criticism 2.

6.5 Matrix3D — 2025.02 (arXiv)/ CVPR 2025

Problem addressed

Matrix3D is the first work explicitly targeting unified reconstruction and generation. A multimodal DiT combines images, camera parameters, and depth in one diffusion process; mask learning permits full-modality training with incomplete data. In one sentence: diffusion-based modality completion turns photogrammetry tasks into one fill-in-the-blank problem.

Unresolved gaps

• Speed: 27s/view and 2228T FLOPs, explicitly acknowledged as too costly for real-time applications.
• Consistency: Attractive images often lack 3D consistency. Unposed CD=0.0670 is 2.58× worse than VGGT's 0.0260 without generation.

How the Core Paper Responds

RnG answers Matrix3D by reversing direction: Matrix3D teaches reconstruction to a diffusion generator; RnG gives a reconstruction model a generative outlet. The resulting gains span CD reduction of 10.0×(exactly an order of magnitude), PSNR +7.540 dB, and speed 317.6×.

Transfer of Key Designs

What transfers is the problem definition and evaluation protocol rather than modules. Matrix3D defines joint evaluation of pose, depth, NVS, and complete 3D; RnG's Table 2 inherits this structure while changing every result. This is critical inheritance: accept a competitor's criteria and outperform it on those terms.

7.1 Nonlinear Evolution Tree

7.2 Underlying Logic

Driver 1: Less Specialized Representations

Each step weakens the premise that 3D requires specially designed structures. RnG takes this furthest: representations are reused rather than designed.

Driver 2: Less Annotation-Dependent Supervision

Driver 3: Reconstruction and Generation Converge, Reversing Direction

Key Turning Points There are two:

1. VGGT's arrival in March 2025 changed the balance. Before it, weak reconstruction models justified adding geometry to generators. Afterward, reconstruction accuracy far exceeded diffusion, making reversal natural. RnG states this explicitly.
2. Invariance→cacheability in March 2026 turned a scientific question into a systems benefit. The causal mask introduced as a control also delivers 5.35× fewer FLOPs. Such alignment between methodological rigor and speed is unusual at the systems/algorithms boundary and worth noting.

Shift in community attention: Initially, Accuracy(PSNR, Chamfer); then Generalization(zero-shot, cross-dataset); now operability of representations : editing, compression, composition, and time. RnG translates these into familiar LLM-system problems—KV compression, cache reuse, and attention pruning—potentially its greatest spillover value.

9.1 Theory

T1 · What Capacity Bound Does a KV-Cache 3D Representation Have?

More specifically than studying expressiveness: For a fixed L-layer × N-token × d-dimensional cache, what scene complexity can be reconstructed losslessly? Fix the backbone and apply low-rank truncation retaining top-r singular values or token pruning retaining k%. Plot compression against CD/PSNR. A sharp knee would indicate information capacity; smooth degradation would suggest redundancy. A high-value, low-barrier experiment requiring only inference, not retraining.

T2 · Is Target-View Invariance Necessary and Sufficient for a Representation?

Table 3's comparable FullAttn quality suggests approximate invariance may emerge without architectural enforcement. Measure reconstruction changes for fixed sources under different target queries. If differences remain below 0.1% relative change, invariance is a property that can be verified afterward rather than imposed beforehand. This could change the interpretation of the core contribution.

T3 · Transfer Limits of Reconstruction Priors for Generation

Is hidden unsupervised geometry in VGGT general or 3D-specific? Apply query interfaces, generative heads, and limited fine-tuning to segmentation (SAM), depth, and detection backbones, testing whether other unsupervised capabilities can be read out.

9.2 Engineering and Applications

E1 · Add the Missing Controls (Criticism 2)

Specific gap: RnG's superiority to LVSM lacks a fair head-to-head. Under identical hardware, data budgets, and initialization compare (i) LVSM from scratch, (ii) LVSM initialized from VGGT, and (iii) RnG. If (ii) matches (iii), gains mainly reflect pretraining rather than architecture. This is the most important next experiment.

E2 · Standardize Decomposed CD Reporting (Criticism 3)

Specific gap: Report a 2×2 matrix of accuracy/completeness and visible/unseen regions. Unseen-region accuracy most honestly measures whether generation is correct. Total CD cannot distinguish accurate completion from simply generating more geometry.

E3 · Missing Scene-Level Benchmarks (Criticism 5)

Specific gap: Build an incomplete-geometry benchmark from handheld indoor video, with ground truth for walls and floors occluded by furniture. Target occluded-region accuracy rather than overall CD; without this, real-scene completeness claims cannot be falsified.

E4 · Cache Policies for Dynamic Online Use

Specific gap: Streaming observations, such as a moving robot, require incremental cache updates. Should new observations append K/V entries or merge with old ones? Appending grows memory linearly; merging loses information. Design admission/eviction policies analogous to H2O or StreamingLLM, using each view's CD improvement as an admission score, and measure CD versus fixed cache budget.

E5 · Cache Editability

Specific gap: First mask source-token K/V corresponding to one object and test whether rendering removes it cleanly instead of creating artifacts. This simple experiment helps decide whether the cache is a representation or merely an acceleration buffer.

9.3 Emerging Directions

N1 · Time: KV-Cache as Explicit World-Model Memory

Specific gap: Video world models rely on external memory mechanisms with mostly heuristic representations. RnG suggests using a reconstruction backbone's KV-cache as spatial memory queried by a video generator each frame. The question is: which matters more for long-term consistency—geometrically consistent or appearance-consistent memory? Compare one generator with (a) RnG-style geometric cache, (b) visual-feature cache, and (c) both, measuring drift after 1,000 frames.

N2 · World-Frame Equivariance (Criticism 6)

Specific gap: Can world coordinates be equivariant to source-view selection? Normalize using predicted point-cloud centroid and principal axes, adding consistency loss under random source-view permutations. Measure CD between reconstructions of the same object from different source subsets; lower means more equivariant. This metric is currently unreported.

N3 · Cache Compression and a Neural 3D Asset Format

Specific gap: If 400MB of K/V compresses to 10MB with CD degradation below 5%, it becomes a transferable 3D asset format supporting rendering through decompression. The research question is: what is the rate–distortion curve of 3D content represented by neural caches? Compare directly with 3DGS and mesh compression.

N4 · Explicit Uncertainty Modeling

Specific gap: RnG's L_pmap already includes aleatoric uncertainty Σ_t, but never exposes it as an output. Could Σ_t reliably indicate observed versus guessed geometry? Split GSO pixels by ground-truth visibility and test separation of Σ_t distributions with AUROC. Success would yield confidence-aware complete 3D at no additional training cost, valuable for physical simulation. The capability may already be trained, just unused.

Gen3R: 3D Scene Generation Meets Feed-Forward Reconstruction (2026.01, v2 2026.03, arXiv)

Also based on VGGT, but couples video diffusion instead of remaining feed-forward: an adapter aligns geometric and appearance latents, generating RGB video and geometry jointly. This represents scene-level generation and matters because VGGT-latent reuse has more than one implementation path.
arXiv ｜ HTML

NOVA3R: Non-pixel-aligned Visual Transformer for Amodal 3D Reconstruction (2026.03, arXiv)

NOVA3R directly tackles the point-map limitation that geometry requires pixels, using a non-pixel-aligned ViT for amodal reconstruction including occluded regions. It and RnG offer two answers to the same problem : RnG supplies pixels through queries; NOVA3R removes the pixel requirement.
arXiv HTML

TRELLIS.2 / Native and Compact Structured Latents for 3D Generation (2025.12, arXiv:2512.14692)

The opposite pole of 3D generation uses native 3D-latent diffusion rather than image reconstruction. It contrasts purpose-built structured latents with reused general-Transformer K/V. Comparing their rate–distortion curves may become a central debate over the next two years.
arXiv ｜ Project page

Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory (2026.04, arXiv:2604.08995)

A real-time streaming interactive world model whose central challenge is the representation of long-term memory, directly related to RnG (§9.3 N1). It demonstrates current engineering limits under joint real-time and long-consistency constraints.
arXiv ｜ Project page

Transition Models: Rethinking the Generative Learning Objective (CVPR 2026 Highlight)

A generative objective accommodating arbitrary transition step counts, turning few-step generation from distillation into objective design. Related question: how much of RnG's 317× advantage over Matrix3D would genuinely few-step diffusion erase?
CVPR 2026 Highlights

tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction (CVPR 2026 Highlight)

Uses test-time training for long-context autoregressive 3D reconstruction, directly relevant to cache scalability (criticism 7): when observations exceed the training configuration, should we expand the cache or use TTT?
CVPR 2026 Highlights

RayZer: A Self-supervised Large View Synthesis Model (ICCV 2025, Best Student Paper HM)

Removes pose and geometry supervision, learning from 2D images with ray structure as the sole 3D prior. Its relevance: RayZer's self-supervised framework is a plausible alternative if RnG is to escape dependence on VGGT pretraining (criticism 2).
arXiv ｜ Code ｜ Project page

LVSM (ICLR 2025 Oral)—Worth Emphasizing Again

It remains strongest on GSO PSNR (27.522), with fully open-source code. Claims of improved novel-view synthesis should compare against LVSM-100K, not only LVSM-40K.
arXiv ｜ Code ｜ ICLR Proceedings