SemoDepth: Selection, Not Fusion — Radar-Modulated State Space Models for Radar-Camera Depth Estimation

SemoDepth architecture: RMS injects radar into the selective scan; MVSP allocates fusion across decoder resolutions.

SemoDepth architecture. (a) Pipeline. A ResNet-34 image encoder and a radar graph structure extractor (GSE) built on PCA-GM produce the image pyramid $c_0,\dots,c_4$ and one radar feature map. Projections ($1{\times}1$) of this map at each level form the radar pyramid $r_0,\dots,r_4$. MVSP allocates fusion by resolution. FiLM modulates the two finest levels (Tier 1). Windowed RMS around each projected radar return serves the middle level (Tier 2). RMS over the full image in four directions serves the two coarsest levels (Tier 3). (b) Radar-Modulated Selection (RMS). Radar adds modulations to the step size $\boldsymbol{\Delta}_t$ and readout $\mathbf{C}_t$ (teal arrows). The input projection $\mathbf{B}_t$ is computed from the image only (blue arrows), and the state matrix $\mathbf{A}$ never sees radar. The state update block applies zero order hold (ZOH) discretization and the recurrence $\mathbf{h}_t = \bar{\mathbf{A}}_t\mathbf{h}_{t-1} + \bar{\mathbf{B}}_t x_t$. The readout block emits $y_t = \mathbf{C}_t^{\top}\mathbf{h}_t$. All radar projections start at zero, so the block starts as vanilla Mamba.

Abstract

Radar camera depth estimation turns a very sparse but metric radar signal into a dense depth map. Existing methods combine radar and image features outside the sequence operator of the backbone, through concatenation, confidence gating, sparse supervision, or graph based extraction. Mamba variants for multimodal fusion also leave the selection mechanism itself unimodal. We argue that selection is the right place for radar to enter.

We introduce Radar-Modulated Selection (RMS), which injects radar inside the selective scan of Mamba. Radar adds learned perturbations, initialised at zero, to the step size $\boldsymbol{\Delta}$ and the readout $\mathbf{C}$. The input projection $\mathbf{B}$ and the state matrix $\mathbf{A}$ never see radar. At initialisation the block equals a pretrained image Mamba, so radar gains influence only where it improves accuracy. The construction also couples the two modalities at linear cost and falls back to the image backbone when radar is missing.

We deploy RMS in a Multi-View Scan Pyramid (MVSP) that matches the fusion operator to the reach of radar at each scale. On nuScenes, SemoDepth reduces MAE by 34.0%, 29.9%, and 29.9% over the previous best within 50, 70, and 80 m. It also has the lowest single frame latency among methods with a public release (26.8 ms). A feature blend placed before the scan adds no accuracy on top of RMS, which supports the view that selection inside the scan can replace fusion outside it. When twice as many spurious radar returns as true ones are injected, the MAE@80 of SemoDepth rises by 1.4%, against 33.6% for a Mamba baseline that fuses radar before the scan.

Key ideas

1. Radar-Modulated Selection (RMS)

Inside each Mamba block, radar features contribute additive, zero-initialised modulations to the per-token step size Δ (memory horizon) and readout C (what the state emits), while the input projection B and state matrix A never see radar. At step zero the block equals a vanilla Mamba scan, so the pretrained image solution is preserved exactly and radar gradients grow only where they improve accuracy. With A and B unimodal, the model falls back to image behaviour under sparse or missing radar.

2. Multi-View Scan Pyramid (MVSP)

A three-tier decoder matches the fusion operator to radar's reach at each resolution: scene-wide four-direction RMS at the coarsest scales, radar-centred windowed RMS at the mid scale, and constant-cost FiLM at the finest scales. Radar stays active at every level, with scan compute concentrated where radar evidence reaches furthest. MVSP lowers MAE@80 by 18% compared with FiLM at every level.

Results on nuScenes

Range	Method	MAE (mm) ↓	RMSE (mm) ↓
0–50 m	TacoDepth (CVPR'25)	1423.6	3275.8
0–50 m	SemoDepth (ours)	940.1 ±4.1	2785.7 ±2.5
0–70 m	TacoDepth (CVPR'25)	1712.6	3960.5
0–70 m	SemoDepth (ours)	1199.8 ±1.5	3622.4 ±16.5
0–80 m	TacoDepth (CVPR'25)	1833.4	4150.2
0–80 m	SemoDepth (ours)	1285.2 ±0.9	3935.4 ±21.5

SemoDepth numbers are mean ± std over 3 seeds. Single-frame latency: 26.8 ms (lowest among methods with public releases). See the arXiv preprint for the full per-method table.

Results on ZJU-4DRadarCam

Range	Method	MAE (mm) ↓	RMSE (mm) ↓
0–50 m	TacoDepth (CVPR'25)	1120.1	2686.7
0–50 m	SemoDepth (ours)	1029.2	2631.6
0–70 m	TacoDepth (CVPR'25)	1181.8	2906.3
0–70 m	SemoDepth (ours)	1111.6	2946.9
0–80 m	TacoDepth (CVPR'25)	1201.1	2990.7
0–80 m	SemoDepth (ours)	1137.2	3053.0

Baseline numbers as reported by TacoDepth. SemoDepth attains the best MAE at every range and the best RMSE@50; TacoDepth retains a slight edge on RMSE@70 (+1.4%) and RMSE@80 (+2.1%).

Robustness to radar corruption

Change in MAE@80 on nuScenes under spurious radar returns, dropped returns, and depth noise, for SemoDepth and FusionMamba.

Change in MAE@80 on nuScenes under radar corruption at test time. (a) Spurious returns added, as a share of the true returns. (b) Share of true returns kept. (c) Gaussian noise on the depth of true returns. FusionMamba† is our reimplementation of FusionMamba, which shares supervision, loss, recipe, and parameter budget with SemoDepth but fuses radar before the scan. Both models see identical corrupted inputs. With radar removed entirely, SemoDepth loses 3.3% (against 155% for FusionMamba†) and still beats every published method on nuScenes.

Qualitative comparisons

Qualitative predictions on nuScenes: RGB, baselines, SemoDepth, accumulated LiDAR ground truth.

nuScenes. Rows show night, day, and rain. Columns show the RGB input, CaFNet, Li et al., Singh et al., and SemoDepth. Each panel gives the MAE and RMSE of the frame in millimetres against sparse LiDAR within 80 m. SemoDepth has the lowest MAE and RMSE on every frame. The largest margin is in rain, with 535 mm MAE against 1915 mm for the next best baseline.

Qualitative predictions on ZJU-4DRadarCam: campus driving scenes.

ZJU-4DRadarCam. Columns show the RGB input, radar returns over the RGB, the SemoDepth prediction, and LiDAR ground truth from a single sweep. The frames were drawn at random from the test split, so the spread of errors reflects the test set rather than selection. SemoDepth recovers vehicle and lane geometry although the radar provides only about 380 returns per frame.

Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation