---
license: mit
---
WildGaussians: 3D Gaussian Splatting in the Wild
Jonas Kulhanek
·
Songyou Peng
·
Zuzana Kukelova
·
Marc Pollefeys
·
Torsten Sattler
We introduce WildGaussians, a novel approach to handle occlusions and appearance changes with 3DGS.
By leveraging robust DINO features and integrating an appearance modeling module within 3DGS, our method achieves state-of-the-art results.
We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines in handling in-the-wild data, all within a simple architectural framework.
## Model Details
- **Developed by:** CTU in Prague, ETH Zurich
- **License:** MIT
- **Repository:** https://github.com/jkulhanek/wild-gaussians
- **Web:** https://wild-gaussians.github.io
- **Paper:** https://arxiv.org/pdf/2407.08447
## Installation
We offer two ways to install the method: local installation (through a conda environment) and installation from [NerfBaselines](https://jkulhanek.com/nerfbaselines), where multiple backends (conda,docker,apptainer) can be selected.
### Install from NerfBaselines (recommended)
The easiest way to install WildGaussians is by using [NerfBaselines](https://jkulhanek.com/nerfbaselines).
First (if not already done), install NerfBaselines by running:
```bash
conda create -n nb python=3.11
conda activate nb
pip install nerfbaselines>=1.0.1
```
WildGaussians is then installed by running:
```bash
nerfbaselines install-method --spec https://raw.githubusercontent.com/jkulhanek/wild-gaussians/main/wildgaussians/wildgaussians_spec.py
```
### Local installation
Clone the repository and create a `python == 3.11` Anaconda environment with CUDA toolkit 11.8 installed using
```bash
git clone git@github.com:jkulhanek/wild-gaussians.git
cd wild-gaussians
conda create -y -n wg python=3.11
conda activate wg
conda install -y --override-channels -c nvidia/label/cuda-11.8.0 cuda-toolkit
conda env config vars set NERFBASELINES_BACKEND=python
pip install --upgrade pip
pip install -r requirements.txt
pip install nerfbaselines>=1.0.1
pip install -e ./submodules/diff-gaussian-rasterization ./submodules/simple-knn
pip install -e .
```
## Checkpoints, predictions, and data
The checkpoints and predictions are available online at [https://huggingface.co/jkulhanek/wild-gaussians/tree/main](https://huggingface.co/jkulhanek/wild-gaussians/tree/main):
In order to train/evaluate on the NeRF On-the-go dataset, please download the undistorted version
from the following link:
[https://huggingface.co/datasets/jkulhanek/nerfonthego-wg/tree/main](https://huggingface.co/datasets/jkulhanek/nerfonthego-wg/tree/main)
## Interactive viewer
To start the viewer and explore the trained models, run one of the following:
```bash
# Photo Tourism
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/trevi-fountain.zip/checkpoint --data external://phototourism/trevi-fountain
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/sacre-coeur.zip/checkpoint --data external://phototourism/sacre-coeur
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/brandenburg-gate.zip/checkpoint --data external://phototourism/brandenburg-gate
# NeRF on-the-go
# NOTE: Optionally attach `--data {path to data}` to load the dataset (recommended)
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/fountain.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/mountain.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/corner.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/spot.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio-high.zip/checkpoint
```
## Training
To start the training on the Photo Tourism dataset, run one of following commands:
```bash
# Photo Tourism
nerfbaselines train --method wildgaussians --data external://phototourism/trevi-fountain
nerfbaselines train --method wildgaussians --data external://phototourism/sacre-coeur
nerfbaselines train --method wildgaussians --data external://phototourism/brandenburg-gate
# NeRF on-the-go
nerfbaselines train --method wildgaussians --data {path to data}
```
The training will also generate output artifacts containing the **test set predictions**, **checkpoint**, and **tensorboard logs**.
## Rendering videos
To render a video on a trajectory (e.g., generated from the interactive viewer), run:
```bash
nerfbaselines render --checkpoint {checkpoint} --trajectory {trajectory file}
```
## Concurrent works
There are several concurrent works that also aim to extend 3DGS to handle in-the-wild data:
- Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections
- Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections
- SpotlessSplats: Ignoring Distractors in 3D Gaussian Splatting
- SWAG: Splatting in the Wild images with Appearance-conditioned Gaussians
- WE-GS: An In-the-wild Efficient 3D Gaussian Representation for Unconstrained Photo Collections
## Acknowledgements
We would like to thank Weining Ren for his help with the NeRF On-the-go dataset and code and Tobias Fischer and Xi Wang for fruitful discussions.
This work was supported by the Czech Science Foundation (GAČR) EXPRO (grant no. 23-07973X)
and by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254).
The renderer is built on [3DGS](https://github.com/graphdeco-inria/gaussian-splatting) and [Mip-Splatting](https://niujinshuchong.github.io/mip-splatting/).
Please follow the license of 3DGS and Mip-Splatting. We thank all the authors for their great work and released code.
## Citation
If you find our code or paper useful, please cite:
```bibtex
@article{kulhanek2024wildgaussians,
title={WildGaussians: 3D Gaussian Splatting in the Wild},
author={Kulhanek, Jonas and Peng, Songyou and Kukelova, Zuzana and Pollefeys, Marc and Sattler, Torsten},
journal={arXiv},
year={2024}
}
``` 
