3DiffGS: 3D Gaussian Splats from Unposed Images using Ray Diffusion

1NAAMII, 2NHU AI, 3INSAIT, 4SUNY Korea

3DiffGS jointly denoises camera rays and regresses 3DGS, and optimizes for better consistency between these two outputs.

Abstract

Recent feed-forward 3D reconstruction methods suffer from (1) a critical supervision bottleneck of requiring expensive 3D data and (2) limited generalization. Our experiments demonstrate that diffusion models yield significantly better generalization; however, such methods exist only for camera pose estimation — not for 3D reconstruction. To enable better generalization and 3D-free training, we introduce 3DiffGS, a camera ray diffusion model that also predicts 3D geometry. Unlike diffusion methods that focus entirely on camera estimation (e.g., PoseDiffusion and RayDiffusion) or depend on explicit 3D data (e.g., DiffusionSfM), 3DiffGS performs diffusion directly in ray space and decodes 3D Gaussian Splats from the same ray features, naturally unifying camera and scene structure prediction. This also enables training without 3D, using only 2D images via re-rendering.

However, we observed that a trivial way of decoding Gaussians and denoised rays leads to inconsistent reconstructions. Therefore, we propose to couple camera ray and geometry predictions using a rendering-based consistency constraint. This encourages structurally coherent prediction of cameras and geometry, while also boosting camera estimation accuracy. 3DiffGS achieves state-of-the-art sparse-view camera pose estimation results among models trained without ground-truth 3D geometry, while utilizing only half the model size of RayDiffusion and demonstrating superior cross-category generalization.

Method

We jointly denoise rays and regress 3D Gaussians using the same features. We also incorporate Rendering Guided Photometric Supervision (RGPS), which encourages predicted cameras to be photometrically consistent with the predicted Gaussians.

Results


Quantitative

3DiffGS generalizes to unseen categories better than RayDiffusion, using just half the parameter count. It performs much better than a joint rays+3DGS regression baseline across camera pose, photometric consistency, and geometry accuracy.


Qualitative

3DiffGS is able to predict accurate camera poses in challenging object-centric scenarios. 3D geometry supervised models (VGGT, DiffusionSfM, AnySplat) perform better in both pose and geometry prediction.

Ablations


Quantitative

RGPS, our training-time strategy for better photometric consistency, not only aligns the predicted cameras and Gaussians better (PSNR), but also results in much more accurate pose prediction. Novel view and mean regularization losses encourage much better geometry by constraining Gaussians better and allocating them better across the observed views.

Qualitative

Without novel view and mean regularization losses, predicted geometry has visible holes and floaters. With RGPS, we observe via RGB renders from predicted cameras that cameras and Gaussians are better aligned.

Limitations

  • There is a significant pose and geometry accuracy gap compared to geometry supervised models such as VGGT and DiffusionSfM.
  • Relying on photometric supervision alone results in overly smooth and blurry reconstruction, especially in more challenging settings (refractive materials, extreme baselines).
  • Our experiments are exclusively on object centric settings.

Video

BibTeX (coming soon)