BridgeVLA: Bridging 3D Robot Inputs and Action Outputs in a Shared 2D Heatmap Space

2.1 Conventional VLAs have strong semantics but weak sample efficiency

VLM-based VLAs inherit web-scale visual–language semantics but typically map images directly to continuous action sequences. Robot data is expensive, yet hundreds of trajectories per task remain common. In contrast, 3D policies such as PerAct, RVT, and RVT-2 use voxels, point clouds, or orthographic projections to frame action positioning as “finding a point in observation space.” Their strong structural priors make few-shot learning more stable, but they lack large VLMs’ open-vocabulary semantics.

The core paper identifies two specific mismatches (Introduction, PDF pp.1–3):

1. Input mismatch. VLM pretraining uses natural 2D images, whereas 3D VLA fine-tuning injects point clouds/per-pixel 3D coordinates, abruptly shifting the input distribution.
2. Output mismatch. VLM pretraining predicts token sequences, while robot fine-tuning directly regresses 3D vectors; actions lack a spatial structure shared with visual tokens.

This is not an abstract concern. In Table 1 (PDF p.6), adding per-pixel 3D position features within the same BridgeVLA framework reduces mean RLBench success from 88.2% to 56.2%, an absolute decline of 32.0 ppand a relative decline of 36.28%; replacing the heatmap head with a Transformer of similar parameter count for direct position regression (309M vs 303M) reduces success from 88.2% to 31.4%, a drop of 56.8 ppand a relative decline of 64.40%. The paper’s central argument is therefore not that more 3D information is always better, but thatadditional information can hurt when it disrupts the pretrained interface。

2.2 Apparent gains in sample efficiency and the real compute cost

In real-robot Table 12 (PDF p.24), with only 3 trajectories per task, BridgeVLA succeeds in 124/130 trials across 13 tasks, or 95.38%≈95.4%; with 10 trajectories, it succeeds in 126/130, or 96.92%≈96.9%. Yet Appendix A (PDF p.16) discloses that RLBench fine-tuning uses 48×H100×20 hours≈960 H100-hours, with COLOSSEUM requiring roughly another 960 H100-hours. “Sample efficiency” thus describes robot-trajectory volume, not training compute efficiency. The abstract also gives a 3-trajectory success rate of 96.8%, inconsistent with 95.4% in the main text and 95.38% in Table 12; this reporting discrepancy is a key audit point in this report.

4.1 Problem definition

The (i)-th demonstration trajectory is

[ _i={l_i,(o^i_1,ai_1),,(o^i_H,ai_H)}, ]

where language instruction is (l), observation (o) consists of one or more RGB-D views, and action (a=(T,g,c)) contains a 6-DoF end-effector pose (TSE(3)), gripper (g{0,1}), and collision flag (c{0,1}) (Sec. 3.1, PDF p.4). The policy predicts the next keyframe, and a motion planner moves to the target pose until success or the maximum step count.

4.2 Object-grounding heatmaps

For the center (x_i^b) of the (i)-th target box, construct a truncated Gaussian (Eq. 2, PDF p.5):

p_i(x)=(-). ]

Average across objects, then normalize (Eq. 3, PDF p.5):

[ H^{gt}(x)=, H_{avg}(x)=_{i=1}^N H_i^{gt}(x). ]

PaliGemma image tokens are rearranged into a 2D grid by patch position, restored to input-image resolution with convex upsampling, and supervised using cross-entropy. The output remains a probability distribution over the image rather than a coordinate vector without spatial structure.

4.3 3D translational actions

Project the point cloud into top/front/right images (I_v); the VLM produces heatmaps (H_v). For a candidate 3D grid point (q) in the workspace, project it into each view (_v(q)); the combined score can be written as

[ s(q)=_{v{top,front,right}} H_v(_v(q)), q=_q s(q). ]

The paper describes backprojecting three views to estimate scores for all 3D grid points (Sec. 3.3, PDF p.5), but provides no separately numbered equation in the main text. The expression above faithfully formalizes its procedure rather than introducing a new algorithm.

4.4 Multi-head action loss

Training objective (Eq. 4, PDF p.6):

[ L=L_{trans}+L_{rot}+L_{gripper}+L_{collision}. ]

Translation and three-axis rotation use cross-entropy; gripper/collision use binary cross-entropy. Position heatmaps provide dense supervision; the remaining discrete heads predict from global/local tokens.

4.5 Pseudocode

Pretraining:
  for (image, text, boxes) in RoboPoint-120K:
      H_gt = normalize(mean(truncated_gaussian(box centers)))
      image_tokens = PaliGemma(image, text)       # SigLIP and token embeddings frozen
      H_pred = ConvexUpsample(reshape(image_tokens))
      optimize CE(H_pred, H_gt)

Robot fine-tuning/inference:
  point_cloud = fuse_calibrated_RGBD(observation)
  views = orthographic_render(point_cloud, top/front/right)
  heatmaps, image_tokens = BridgeVLA(views, instruction)
  translation = argmax_3D(backproject_and_fuse(heatmaps))
  local = tokens_at_heatmap_peaks(image_tokens)
  global = max_pool_each_view(image_tokens)
  rotation_3x72, gripper, collision = MLP(concat(local, global))
  planner.move_to(translation, rotation, collision)
  repeat until oracle success or H_max

6.1 RLBench

18 tasks, each with 100 expert demonstrations; binary success per task, 25 trials/task, and at most 25 action steps. The authors train BridgeVLA five times (Sec. 4.1.1, PDF p.7). Main metrics are mean success rate and mean rank over 18 tasks. It tests complex manipulation, but training and testing use the same objects, so open-world category generalization is not its focus.

6.2 COLOSSEUM

Extends RLBench to 20 base tasks and 12 types of unseen perturbations, totaling 20,371 task-perturbation instances; also includes original RLBench variations and an “All Perturbations” setting. Each perturbation is evaluated on 3 unseen instances, with 25 trials per task. BridgeVLA/RVT-2 are retrained three times; other baselines come from the benchmark paper (Sec.4.1.2 pp.7–8; App.B.3 p.16).

6.3 GemBench

Training uses 16 tasks/31 variations and seven basic action types; testing covers 44 tasks/92 variations, divided into L1 new positions, L2 new rigid objects, L3 new articulated objects, and L4 new long-horizon compositional tasks. BridgeVLA uses 5 seeds, each with 20 trials per variation (Sec.4.1.3 pp.8–9; App.B.4 p.16).

6.4 Real robot

Franka Research 3 + ZED 2i, 13 tasks, 10 expert trajectories per task; seven settings: Basic, Distractor, Lighting, Background, Height, Combination, Category. Table 4 Basic uses 10 trials per method per task; the remaining six settings compare only RVT-2, BridgeVLA without pretraining, and full BridgeVLA (Sec.4.2 pp.9–11).

7.1 Overall results on the three main benchmarks

7.2 GemBench: complete matrix of five metrics × seven methods

All values are success percentages; ± values are copied from the original table, which does not specify whether they are standard deviations or standard errors. L4 contains the hardest long-horizon compositional tasks.

Source: Table 3, PDF p.9. BridgeVLA’s highest overall mean is driven mainly by L2/L3; it is not best on L1, and L4 is 0.0, behind 3D-LOTUS++ at 17.4 pp. Thus, “overall SOTA” does not imply leading long-horizon planning capability.

7.3 COLOSSEUM: complete matrix of 14 perturbation settings

Source: Table 2, PDF p.8. BridgeVLA ranks first in 13 of 14 settings; its only clear miss is Distractor:51.8 vs RVT-2’s 60.8, lower by 9.0 pp (−14.80%). The authors emphasize 13/14, but this failure suggests that heatmaps can still ground incorrectly in the presence of similar distractors.

7.4 RLBench method lineage and ablations

Source: BridgeVLA Table 1 p.6 and Appendix A p.16; earlier speeds were also read from RVT/RVT-2 Table 1/I. BridgeVLA has the highest success rate, but its inference frequency is below RVT-2’s 20.6fps by approximately 76.9%; moreover, its 20h requires 48 H100s, so “0.83 days” must not obscure the hardware scale.

7.5 Real-robot Basic setting

13 tasks × 10 trials：SpatialVLA(50 demos) 28.5%，SpatialVLA(10) 3.1%，π0 3.8%，ACT 22.3%，RVT-2 90.0%，BridgeVLA 96.9%(Table 4, PDF p.10). BridgeVLA exceeds RVT-2 by +6.9 pp and SpatialVLA, also using 10 demos, by +93.8 pp. However, the original SpatialVLA paper pretrains on 1.1M episodes and evaluates different action representations/platforms. Whether its fine-tuning configuration is optimal when transferred here is insufficiently documented, so the gap measures both the method and reproduction recipe.

8.1 Output space: heatmaps provide the strongest causal ablation

A direct coordinate-regression head with similar parameter count reduces RLBench success from 88.2% to 31.4%, namely −56.8 pp; the authors also report that direct regression requires increasing batch size to 192 and carefully searching learning rates, whereas the heatmap version trains stably at batch 64 (Sec.4.3, PDF p.11). This supports three mechanisms: denser heatmap supervision, spatial priors from 3D→2D projection, and a shared 2D topology between inputs and outputs.

8.2 Input space: more 3D information can disrupt the VLM

Adding a 3D convolution to encode per-pixel positions reduces success from 88.2 to 56.2 (−32.0 pp). The paper attributes this to changes in the pretrained image-feature distribution. This ablation matters, but intermediate settings such as normalization, adapter-only fusion, and frozen/unfrozen ratios are missing. It therefore shows that this fusion scheme is harmful, not that every explicit 3D feature is harmful.

8.3 Pretraining: semantic-generalization gains shown only in a chart

Fig.3 (PDF p.11) shows that removing 2D heatmap pretraining substantially lowers Combination and Category performance, even below RVT-2. Visually, the full-model Combination bar is around 0.64, while the no-pretraining bar is near 0.04. Because exact value labels are absent, this report does not treat pixel estimates as paper-reported numbers. The figure supports the importance of pretraining for semantic OOD, but not a reliable pp gain or error-bar calculation.

8.4 Data-volume phase transition and inconsistent internal reporting

In Table 12 (p.24), 3 demos yield 124/130=95.38%, and 10 demos yield 126/130=96.92%, only 1.54 pp apart; the main difference is wolf-upper-drawer (7/10→9/10). Yet the abstract gives 96.8% for 3 demos, the main text 95.4%, and the table recalculates to 95.38%. Therefore, “3 demos reach 96.8%” should not be copied; the defensible conclusion is approximately 95.4% from appendix data.

8.5 Hyperparameter sensitivity: no systematic curves

Table 5 gives only final settings: pretraining lr 5e-5/batch 384; RLBench/COLOSSEUM/GemBench lr 8e-5 with batch 192/192/160; real robot lr2e-5/batch 192. No systematic sweep is reported for (,p_{min}), grid resolution, orthographic view count, 36/144 alternatives to 72 bins, convex-upsampling depth, frozen-layer ratio, or data scale. The direct-regression variant reportedly underwent grid search, but neither the grid nor failed points is published, preventing assessment of whether 31.4% is a fair optimum.

10.1 PerAct: treating actions as 3D voxel detection

Problem solved. PerAct uses voxelized RGB-D and a Perceiver Transformer to classify the next action within a 3D grid; 18 RLBench tasks ×249 variations, 25 episodes/task. Original Table 1 (PDF p.7) reports mean gains over C2FARM-BC of 1.33× with 10 demos and 2.83× with 100 demos.

Remaining gap. Voxel computation is expensive; a later standardized reevaluation reports mean success of 49.4%, 16 days of training, and 4.9fps (RVT Table 1, PDF p.5). Precision tasks insert peg/stack cups/place cups are near zero.

How BridgeVLA responds. Retains the principle of inputs and actions sharing one space, but replaces 3D voxels with three 2D heatmaps and adds VLM semantics.

Inherited design. Keyframe extraction rules, 72-bin rotation classification, and gripper/collision heads are directly inherited from PerAct. BridgeVLA’s novelty is interface alignment rather than redefining the control problem.

10.2 RVT: orthographic views replace heavy voxels

Problem solved. RVT renders virtual views from point clouds and predicts 3D translation with a multi-view Transformer and heatmap backprojection. RLBench success is 62.9%, a 27.33% relative gain over PerAct’s 49.4%; training takes 1 day versus 16 days, with inference at 11.6 versus 4.9fps (RVT Table 1, PDF p.5).

Remaining gap. Fine insertion remains weak: insert peg 11.2±3.0, place cups 4.0±2.5; views are manually chosen and open-ended semantics are absent.

How BridgeVLA responds. Almost directly adopts top/front/right projection and heatmap backprojection, then upgrades the visual processor to PaliGemma and adds text-conditioned heatmap pretraining.

Inherited design. BridgeVLA’s “2D bridge” does not emerge from nowhere: its geometric framework comes from RVT. The actual addition turns that framework into a pretraining/fine-tuning interface compatible with a VLM.

10.3 RVT-2: jointly improving precision and systems

Problem solved. RVT-2 adds location-conditioned rotation, convex upsampling, mixed precision, and other improvements, raising RLBench success from RVT’s 62.9% to 81.4%, with 0.83-day training and 20.6fps (RVT-2 Table I, PDF p.6). Insert peg rises from 11.2 to 40.0, and place cups from 4.0 to 38.0.

Remaining gap. It remains a non-VLA specialist with limited semantics and OOD instruction capability; the paper itself notes that it observes only the current moment, lacks history, and struggles with long tasks (RVT-2 limitations, PDF p.8).

How BridgeVLA responds. Reuses convex upsampling and multi-view heatmaps, placing PaliGemma’s language-conditioned image tokens into the same structure.

Inherited design. BridgeVLA’s 81.4 RLBench baseline is RVT-2. Its central 6.8 pp gain resembles adding a semantic backbone to an efficient 3D policy more than replacing all control modules.

10.4 SpatialVLA: a VLA route with explicit 3D position injection

Problem solved. SpatialVLA uses Ego3D Position Encoding and Adaptive Action Grids, pretraining on 1.1M real-world episodes. LIBERO Table III (PDF p.7) reports a mean of 78.1±0.7%, above fine-tuned OpenVLA’s 76.5±0.6; L1 Spatial is 88.2±0.5.

Remaining gap. Explicit 3D position tokens alter VLM image representations, while actions still use tokens/grids. Transferred to BridgeVLA’s real tasks with 10 demos, Basic success is only 3.1%, and even 50 demos reach only 28.5% (BridgeVLA Table 4, p.10).

How BridgeVLA responds. Takes the opposite choice of avoiding 3D coordinate injection into the backbone and using image-like orthographic projections instead; Table 1’s w/pos ablation directly shows 88.2→56.2.

Inherited design. Both view spatial representation as central to VLA, but differ at the interface: SpatialVLA makes the VLM accept 3D; BridgeVLA re-encodes 3D into the VLM’s native 2D form.

10.5 Nonlinear technical evolution

                     ┌─ VLM / web-scale semantics ── OpenVLA, π0, SpatialVLA
                     │                                  │
PerAct: voxel Q-map ─┼─► Action detection in a shared space ├─ Ego3D + action tokens
    │                │                                  │       ↓ Input distribution shift
    │                └─► RVT: 3D→multi-view 2D heatmap ─┼─► BridgeVLA
    │                           │                        │   PaliGemma image tokens
    │                           ▼                        │   + object heatmap pretrain
    └──────────────► RVT-2: convex upsampling/precision/systems ┘   + 3-view backprojection
                                 │
                                 └─ External motion planner / keyframe execution

The driving concern shifts from improving sample efficiency with 3D grids, to reducing 3D computation, to incorporating pretrained VLM semantics. The real turning point is not a larger model, but recognizing thatthe representation interface itself is an inductive bias: directly injecting the same 3D information can disrupt pretraining, whereas projection–heatmap bridging can reuse existing representations.

10.6 Cross-paper comparison

11.1 Actual contributions

1. Interface alignment rather than module accumulation. BridgeVLA’s key is simultaneously aligning inputs (3D→2D) and outputs (actions→2D heatmaps), rather than adding another 3D encoder.
2. Pretraining objective and downstream head share a structure. Language-conditioned object grounding first trains heatmap prediction, avoiding a sudden downstream requirement for the VLM to emit an unfamiliar structure.
3. Systematic evidence. Three simulation benchmarks +13 real tasks show gains beyond RLBench alone; structural ablations span 32–56.8 pp, far exceeding the main-leaderboard gains of 2–7.3 pp.
4. Counterintuitive finding. Explicit 3D position features hurt performance, showing that incorporating new modalities into large models depends on distributional compatibility, not monotonically increasing information.

11.2 Reusable designs

• For any pretrained backbone, first translate a new modality into a familiar token topology, then solve the downstream task; this is more stable than directly concatenating coordinates.
• Output heads should ideally share spatial structure with inputs: video actions can use spatiotemporal tube heatmaps, navigation bird’s-eye occupancy/action maps, and web agents screenshot-element heatmaps.
• Pretraining proxy tasks should match downstream decoder structure. If the downstream task needs keypoints, first train open-vocabulary keypoint/grounding tasks rather than only next-token prediction.
• When diagnosing harmful new features, compare equal-parameter adapters, frozen-layer ratios, and feature-distribution shifts rather than only presence versus absence.

11.3 Transfer potential

For causal prediction in long videos, future actionable locations could be predicted as spatiotemporal heatmaps, followed by structured event heads. For multimodal moderation representations, high-level risk semantics in UE/AE could be aligned with frame/token saliency maps rather than simply concatenating every vector. For agent GUI interaction, text-conditioned heatmaps naturally match click coordinates. However, 2D bridging has limits for true 3D topology, occlusion, and contact dynamics. Ultimately, a VLM-compatible 2D backbone may need a small 3D residual module that preserves its distribution.

14.1 Theory

• Measuring alignment. Fix PaliGemma and training data; compare raw point tokens, Ego3D, three-view RGB, and three-view RGB plus low-rank 3D residuals. Measure layerwise CKA/MMD relative to the original pretrained image distribution and regress RLBench success to test whether closer distributional matching improves performance.
• Mechanism of dense heatmap supervision. Compare equal-parameter networks using coordinate MSE, discrete 3D-grid CE, 2D Gaussian-heatmap CE, and optimal-transport heatmaps. Report gradient SNR, sample-complexity curves, and final SR to separate topological alignment from loss density.
• Sufficiency of projected views. Control view count at 1/3/6/adaptive; measure occlusion and conditional mutual information, particularly on Place Cups/Insert Peg, to test whether a visibility ceiling determines failures.

14.2 Engineering applications

• Data efficiency at equal GPU-hours. Give BridgeVLA, RVT-2, and SpatialVLA the same 8×H100 hardware and wall-clock budget, with 1/3/10/50 trajectories. Report SR, GPU-hours, peak memory, and end-to-end latency including the planner to obtain a Pareto frontier.
• Interface ablation with a shared backbone. Attach Ego3D, a point adapter, and orthographic heatmaps to PaliGemma with identical frozen/unfrozen ratios. This fairly tests whether BridgeVLA benefits from its backbone or interface.
• Closed-loop perturbations. Move targets during motion and introduce collisions and depth noise; compare keyframe+planner with diffusion action chunks. Report recovery time and failure modes to test whether 0.21s network inference suffices for closed-loop control.

14.3 Emerging directions

• History and long horizons. On GemBench L4, compare memoryless BridgeVLA, a token cache, JEPA-style state prediction, and an LLM task planner while fixing the low-level policy. Add subgoal completion and error recovery to success metrics.
• Open-category action-keypoint pretraining. Compare box-center grounding, affordance keypoints, human hand–object contact, and synthetic robot keypoints, focusing on Category OOD to test the authors’ observation that object centers differ from manipulation keypoints.
• Uncertainty-based routing. Use heatmap entropy, cross-view consistency, and ensemble variance to trigger help requests, view changes, or replanning. Compare head-to-head with always executing the policy, reporting safety failures and added latency.