Recent feed-forward 3D reconstruction methods suffer from (1) a critical supervision bottleneck of requiring expensive 3D data and (2) limited generalization. Our experiments demonstrate that diffusion models yield significantly better generalization; however, such methods exist only for camera pose estimation — not for 3D reconstruction. To enable better generalization and 3D-free training, we introduce 3DiffGS, a camera ray diffusion model that also predicts 3D geometry. Unlike diffusion methods that focus entirely on camera estimation (e.g., PoseDiffusion and RayDiffusion) or depend on explicit 3D data (e.g., DiffusionSfM), 3DiffGS performs diffusion directly in ray space and decodes 3D Gaussian Splats from the same ray features, naturally unifying camera and scene structure prediction. This also enables training without 3D, using only 2D images via re-rendering.
However, we observed that a trivial way of decoding Gaussians and denoised rays leads to inconsistent reconstructions. Therefore, we propose to couple camera ray and geometry predictions using a rendering-based consistency constraint. This encourages structurally coherent prediction of cameras and geometry, while also boosting camera estimation accuracy. 3DiffGS achieves state-of-the-art sparse-view camera pose estimation results among models trained without ground-truth 3D geometry, while utilizing only half the model size of RayDiffusion and demonstrating superior cross-category generalization.