GeoScaffold: Learning Compact Geometric Latents via Reconstruction for Efficient Vision-Language Navigation

Yixuan Jiang1,2, Wentong Li3, An Liu2, Zihao Xin3, Fulin Tang2,4,✉, Cong Leng4, Yang Gao1, Jian Cheng2
1 Nanjing University, Nanjing, China 2 Institute of Automation, Chinese Academy of Sciences, Beijing, China 3 Nanjing University of Aeronautics and Astronautics, Nanjing, China 4 MAICRO, Nanjing, China
Paper

Abstract

Recent vision-and-language navigation (VLN) systems increasingly adopt streaming Video-LLM policies that map egocentric RGB observations and instructions directly to low-level actions. Yet these policies inherit weak 3D geometric priors from 2D pretraining. Existing geometry-aware extensions charge a persistent inference-time price: depth sensors, 3D encoders, or per-step perception tool calls. We propose GeoScaffold, a geometric supervision framework that pays this price once, at training time, by internalizing geometry into the policy itself. It first learns a compact depth tokenizer on depth maps from the training trajectories and freezes it. It then fine-tunes the policy with a handful of learnable geometry query tokens, training their hidden states to reconstruct navigation-critical geometry such as depth, connectivity, and traversability. This supervision turns the query states into compact geometric latents for action decoding, and through the shared weights also internalizes geometry into the backbone's own representations. Like a scaffold, the tokenizer, target generators, and reconstruction heads are discarded after training, leaving the backbone and action interface unchanged. Extensive experiments show that GeoScaffold consistently outperforms leading vision-only navigators on continuous VLN benchmarks, offering a practical paradigm for lightweight edge deployment of spatially aware embodied navigation models.

GeoScaffold training framework with depth, connectivity, and traversability reconstruction heads, plus the hybrid attention mask.
Overview of GeoScaffold. Eight geometry query tokens learn to reconstruct depth, connectivity, and traversability through training-only heads. Their hidden states become compact geometric latents for action decoding. The auxiliary modules are discarded after training.

Demos

Learning Geometry During Training

Three navigation paradigms: direct RGB streaming Video-LLM, a policy with a resident spatial encoder, and GeoScaffold with training-only geometry supervision.
Geometry as a training scaffold. Streaming RGB policies receive geometric supervision through dedicated query tokens, enabling spatial reasoning without a resident depth or 3D encoder.

Stage I: learn a compact depth tokenizer. A 0.74M-parameter VQ-VAE is trained offline on depth maps from navigation training trajectories, then frozen. It represents each depth map as 100 discrete tokens on a 10 × 10 grid, using a 128-entry codebook.

Stage II: reconstruct geometry through query tokens. Eight learnable query tokens are inserted before each action block: four for depth, two for connectivity, and two for traversability. Training-only heads reconstruct the corresponding geometric targets. A hybrid attention mask lets the queries within a turn attend to one another bidirectionally while the remaining sequence stays causal.

Deployment: retain the geometric latents. The tokenizer, target generators, and reconstruction heads are discarded after training. The deployed navigator takes RGB observations, language, and dialogue history. The learned query embeddings and hybrid attention mask remain, allowing action decoding to read the compact geometric latents.

An RGB observation with reconstructed depth, connectivity, and traversability targets decoded from query-token hidden states.
Navigation-relevant geometric targets. Depth encodes metric structure, connectivity highlights passages and openings, and traversability identifies free-space and floor-like regions.

Experimental Results

Original Table 1 from the GeoScaffold manuscript, including its caption, all compared methods, observation requirements, and R2R-CE and RxR-CE Val-Unseen metrics.

Inference Efficiency

Steady-state inference on one NVIDIA RTX 4090
MethodAverage latency ↓
JanusVLN1,160 ms / frame
GA-VLN750 ms / frame
GeoScaffold-4B81 ms / frame

Latency measures the time to process one incoming frame and emit an action, after the streaming context fills its sliding window. GeoScaffold-4B supports over 12 Hz on the RTX 4090. JanusVLN retains a VGGT 3D encoder, and GA-VLN builds a geometry-aware BEV representation from RGB-D observations; neither additional module is needed by GeoScaffold at inference.

Per-frame latency over 80 frames on an RTX 4090 for GeoScaffold-4B, JanusVLN, and GA-VLN. GeoScaffold stays below 100 milliseconds except for periodic history-refresh spikes.
Per-frame latency traces on an RTX 4090. GeoScaffold’s short periodic spikes coincide with updates to the streaming context. The plotted traces complement the steady-state averages above.

BibTeX

@misc{jiang2026geoscaffoldlearningcompactgeometric,
  title={GeoScaffold: Learning Compact Geometric Latents via Reconstruction for Efficient Vision-Language Navigation},
  author={Yixuan Jiang and Wentong Li and An Liu and Zihao Xin and Fulin Tang and Cong Leng and Yang Gao and Jian Cheng},
  year={2026},
  eprint={2610.02697},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2610.02697},
}