README.md

---
license: mit
---
<p align="center">
<h1 align="center">WildGaussians: 3D Gaussian Splatting in the Wild</h1>
  <p align="center">
    <a href="https://jkulhanek.com/">Jonas Kulhanek</a>
    ·
    <a href="https://pengsongyou.github.io/">Songyou Peng</a>
    ·
    <a href="https://cmp.felk.cvut.cz/~kukelova/">Zuzana Kukelova</a>
    ·
    <a href="https://people.inf.ethz.ch/marc.pollefeys/">Marc Pollefeys</a>
    ·
    <a href="https://tsattler.github.io/">Torsten Sattler</a>
  </p>
  <h3 align="center"><a href="https://arxiv.org/pdf/2407.08447">📄 Paper</a> | <a href="https://wild-gaussians.github.io/">🌐 Project Page</a> | <a href="https://github.com/jkulhanek/wild-gaussians">🖥 Code</a></a></h3>
</p>
<p align="center">
  <table>
    <tr>
      <td width="10%"></td>
      <td width="40.8%">
        <img width="100%" alt="WildGaussians model appearance" src="https://github.com/jkulhanek/wild-gaussians/raw/main/.assets/cover-trevi.webp" />
      </td>
      <td width="34.4%">
          <img width="100%" alt="WildGaussians remove occluders" src="https://github.com/jkulhanek/wild-gaussians/raw/main/.assets/cover-onthego.webp" />
      </td>
      <td width="10%"></td>
    </tr>
  </table>
</p>
<p align="justify">
We introduce WildGaussians, a novel approach to handle occlusions and appearance changes with 3DGS.
By leveraging robust DINO features and integrating an appearance modeling module within 3DGS, our method achieves state-of-the-art results.
We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines in handling in-the-wild data, all within a simple architectural framework.
</p>
<br>

## Model Details
- **Developed by:** CTU in Prague, ETH Zurich
- **License:** MIT
- **Repository:** https://github.com/jkulhanek/wild-gaussians
- **Web:** https://wild-gaussians.github.io
- **Paper:** https://arxiv.org/pdf/2407.08447

## Installation
We offer two ways to install the method: local installation (through a conda environment) and installation from [NerfBaselines](https://jkulhanek.com/nerfbaselines), where multiple backends (conda,docker,apptainer) can be selected.

### Install from NerfBaselines (recommended)
The easiest way to install WildGaussians is by using [NerfBaselines](https://jkulhanek.com/nerfbaselines).
First (if not already done), install NerfBaselines by running:
```bash
conda create -n nb python=3.11
conda activate nb
pip install nerfbaselines>=1.0.1
```

WildGaussians is then installed by running:
```bash
nerfbaselines install-method --spec https://raw.githubusercontent.com/jkulhanek/wild-gaussians/main/wildgaussians/wildgaussians_spec.py
```

### Local installation
Clone the repository and create a `python == 3.11` Anaconda environment with CUDA toolkit 11.8 installed using
```bash
git clone git@github.com:jkulhanek/wild-gaussians.git
cd wild-gaussians

conda create -y -n wg python=3.11
conda activate wg
conda install -y --override-channels -c nvidia/label/cuda-11.8.0 cuda-toolkit
conda env config vars set NERFBASELINES_BACKEND=python
pip install --upgrade pip
pip install -r requirements.txt
pip install nerfbaselines>=1.0.1
pip install -e ./submodules/diff-gaussian-rasterization ./submodules/simple-knn
pip install -e .
```

## Checkpoints, predictions, and data
The checkpoints and predictions are available online at [https://huggingface.co/jkulhanek/wild-gaussians/tree/main](https://huggingface.co/jkulhanek/wild-gaussians/tree/main):
<ul>
<li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/trevi-fountain.zip?download=true">Photo Tourism/Trevi Fountain</a></li>
<li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/sacre-coeur.zip?download=true">Photo Tourism/Sacre Coeur</a></li>
<li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/brandenburg-gate.zip?download=true">Photo Tourism/Brandenburg Gate</a></li>
<br>
<li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/fountain.zip?download=true">NeRF On-the-go/Fountain</a></li>
<li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/mountain.zip?download=true">NeRF On-the-go/Mountain</a></li>
<li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/corner.zip?download=true">NeRF On-the-go/Corner</a></li>
<li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio.zip?download=true">NeRF On-the-go/Patio</a></li>
<li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/spot.zip?download=true">NeRF On-the-go/Spot</a></li>
<li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio-high.zip?download=true">NeRF On-the-go/Patio High</a></li>
</ul>

In order to train/evaluate on the NeRF On-the-go dataset, please download the undistorted version
from the following link:
[https://huggingface.co/datasets/jkulhanek/nerfonthego-wg/tree/main](https://huggingface.co/datasets/jkulhanek/nerfonthego-wg/tree/main)

## Interactive viewer
To start the viewer and explore the trained models, run one of the following:
```bash
# Photo Tourism
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/trevi-fountain.zip/checkpoint --data external://phototourism/trevi-fountain
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/sacre-coeur.zip/checkpoint --data external://phototourism/sacre-coeur
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/brandenburg-gate.zip/checkpoint --data external://phototourism/brandenburg-gate

# NeRF on-the-go
# NOTE: Optionally attach `--data {path to data}` to load the dataset (recommended)
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/fountain.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/mountain.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/corner.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/spot.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio-high.zip/checkpoint
```

## Training
To start the training on the Photo Tourism dataset, run one of following commands:
```bash
# Photo Tourism
nerfbaselines train --method wildgaussians --data external://phototourism/trevi-fountain
nerfbaselines train --method wildgaussians --data external://phototourism/sacre-coeur
nerfbaselines train --method wildgaussians --data external://phototourism/brandenburg-gate

# NeRF on-the-go
nerfbaselines train --method wildgaussians --data {path to data}
```

The training will also generate output artifacts containing the **test set predictions**, **checkpoint**, and **tensorboard logs**.

## Rendering videos
To render a video on a trajectory (e.g., generated from the interactive viewer), run:
```bash
nerfbaselines render --checkpoint {checkpoint} --trajectory {trajectory file}
```

## Concurrent works
There are several concurrent works that also aim to extend 3DGS to handle in-the-wild data:
<ul>
<li>Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections</li>
<li>Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections</li>
<li>SpotlessSplats: Ignoring Distractors in 3D Gaussian Splatting</li>
<li>SWAG: Splatting in the Wild images with Appearance-conditioned Gaussians</li>
<li>WE-GS: An In-the-wild Efficient 3D Gaussian Representation for Unconstrained Photo Collections</li>
</ul>
</p>
</section>

## Acknowledgements
We would like to thank Weining Ren for his help with the NeRF On-the-go dataset and code and Tobias Fischer and Xi Wang for fruitful discussions.
This work was supported by the Czech Science Foundation (GAČR) EXPRO (grant no. 23-07973X)
and by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254).
The renderer is built on [3DGS](https://github.com/graphdeco-inria/gaussian-splatting) and [Mip-Splatting](https://niujinshuchong.github.io/mip-splatting/).
Please follow the license of 3DGS and Mip-Splatting. We thank all the authors for their great work and released code.

## Citation
If you find our code or paper useful, please cite:
```bibtex
@article{kulhanek2024wildgaussians,
  title={WildGaussians: 3D Gaussian Splatting in the Wild},
  author={Kulhanek, Jonas and Peng, Songyou and Kukelova, Zuzana and Pollefeys, Marc and Sattler, Torsten},
  journal={arXiv},
  year={2024}
}
```