3D Point Splatting for mmWave Radar

3D Point Splatting (3DPS) enables novel view synthesis for mmWave radar by deriving rendering directly from the radar equation. This approach moves beyond the existing Optical-NVS ports of NeRF, hash grids, and 3D Gaussians, which discard phase information in favor of learned, opaque intensity features. 3DPS instead handles the complex-valued outputs necessary for accurate radar simulation by maintaining the signal's phase and amplitude throughout the rendering process.

Moving from Intensity to Physics

Each 3D point in the scene carries an ITU-R P.2040 material model. The dielectric constant and conductivity of the material determine the complex impedance at the surface. By computing the Fresnel reflection coefficient as a closed-form function of the angle of incidence and the material's permittivity, 3DPS calculates the ratio of the reflected field to the incident field directly. This geometric calculation replaces the iterative ray-tracing required by traditional electromagnetic solvers, allowing the system to determine the exact complex reflection coefficient for any incident wave without performing thousands of numerical approximations.

To map these points into radar data, 3DPS uses a precomputed point spread function (PSF). If you imagine a single metal sphere in a parking lot, the PSF models the radar's response as a spatial distribution across range, azimuth, and elevation bins. The system calculates the response for each bin by convolving the target's projection with the sinc function derived from the radar's physical bandwidth and aperture dimensions. For a target at a specific 3D coordinate, the system projects the point into the radar's spherical coordinate frame, determining its distance and angle of arrival. This transformation yields the round-trip travel time, which defines the phase shift relative to the carrier frequency. Each splat contributes a phasor—a complex number representing magnitude and phase—to the surrounding range bins based on the energy fall-off defined by the PSF window. Because radar signals are coherent, the final bin value is the complex sum of all phasors falling within that volume, allowing constructive and destructive interference to emerge naturally.

Performance and Scaling

Because the output is a coherent field, the scene model stores complex reflections at the point level, meaning the final radar data is generated through an FFT-based post-process. This architecture allows you to simulate different radar bandwidths or receiver configurations using the same trained point cloud without retraining. 3DPS achieves a 0.587 mean Pearson correlation on held-out ColoRadar scenes, which is up to 5.2x higher than the baselines. The computational cost is practical, with scene training requiring approximately 3 minutes on an RTX 4090.

This method improves fidelity but assumes a static environment and requires high-quality surface normals for accurate reflection modeling. When scene geometry or material properties are noisy, phase accuracy degrades because the model cannot compensate for multipath effects that are not explicitly captured in the geometry. The reliance on precomputed PSF remains a constraint when moving from simulated sensor parameters to the non-ideal, hardware-specific responses of real-world transceivers, where beam patterns deviate from ideal sinc functions.