GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting

Kai Zhang*1, Sai Bi*1, Hao Tan*1, Yuanbo Xiangli2, Nanxuan Zhao1, Kalyan Sunkavalli1, Zexiang Xu1 *(Equal contribution)   
1Adobe Research    2Cornell University   

arXiv Cite


High-quality 3D Gaussian primitives from 2-4 posed sparse images within 0.23 second

(This webpage contains a lot of videos and interactive viewers. We suggest using Chrome or Edge for the best experience)


Abstract

We propose GS-LRM, a scalable large reconstruction model that can predict high-quality 3D Gaussian primitives from 2-4 posed sparse images in 0.23 seconds on single A100 GPU. Our model features a very simple transformer-based architecture; we patchify input posed images, pass the concatenated multi-view image tokens through a sequence of transformer blocks, and decode final per-pixel Gaussian parameters directly from these tokens for differentiable rendering. In contrast to previous LRMs that can only reconstruct objects, by predicting per-pixel Gaussians, GS-LRM naturally handles scenes with large variations in scale and complexity. We show that our model can work on both object and scene captures by training it on Objaverse and RealEstate10K respectively. In both scenarios, the models outperform state-of-the-art baselines by a wide margin. We also demonstrate applications of our model in downstream 3D generation tasks.

pipeline

Figure 2. Our simple transformer-based GS-LRM predicts 3D Gaussian parameters from sparse posed images. Images are patchified and the concatenated patch tokens are sent to the transformer blocks. By unpatchifying the transformer's output, each pixel is unprojected to a 3D Gaussian. The final output merges all 3D Gaussians. (Note that here we visualize the Gaussian centers and colors as point clouds for illustration)

Results on Google Scanned Objects

(Click to see more results)

Results on Amazon Berkeley Objects

(Click to see more results)

Results on RealEstate10K

More results: page 1, page 2

Applications: text/image to 3D

Notes: we use the following multi-view generators:
1) Instant3D for object text-to-3D; 2) Zero123++ for object image-to-3D; 3) Sora for scene image-to-3D.
Open interactive viewer
A dog made of vegetables
(Images courtesy of Instant3D)
Open interactive viewer
A brightly colored mushroom growing on a log
(Images courtesy of Instant3D)
Open interactive viewer
A bear dressed in medieval armor
(Images courtesy of Instant3D)
input images
Open interactive viewer
Drone view of waves crashing against the rugged cliffs along Big Sur's garay point beach
(Images courtesy of Sora)
input images
Open interactive viewer
Tour of an art gallery with many beautiful works of art in different styles
(Images courtesy of Sora)

BibTeX

@article{gslrm2024,
    author={Zhang, Kai and Bi, Sai and Tan, Hao and Xiangli, Yuanbo and Zhao, Nanxuan 
      and Sunkavalli, Kalyan and Xu, Zexiang},
    title     = {GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting},
    journal   = {European Conference on Computer Vision},
    year      = {2024},
}