GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting

Yiwen Chen*1,2, Zilong Chen*3,5, Chi Zhang2, Feng Wang3, Xiaofeng Yang2,
Yikai Wang3, Zhongang Cai4 Lei Yang4 Huaping Liu3 Guosheng Lin**1,2
*Equal contribution.
**Corresponding author.
1S-Lab, Nanyang Technological University, 2School of Computer Science and Engineering, Nanyang Technological University, 3Department of Computer Science and Technology, Tsinghua University,
4SenseTime Research,
5ShengShu
Paper arXiv Code Video

GaussianEditor provides controllable, diverse, and interactive high-resolution 3D editing, needing only 2-7 minutes and 10-20GB of GPU memory on a single A6000 GPU.

Abstract

3D editing plays a crucial role in many areas such as gaming and virtual reality. Traditional 3D editing methods, which rely on representations like meshes and point clouds, often fall short in realistically depicting complex scenes.

On the other hand, methods based on implicit 3D representations, like Neural Radiance Field (NeRF), render complex scenes effectively but suffer from slow processing speeds and limited control over specific scene areas. In response to these challenges, our paper presents GaussianEditor, an innovative and efficient 3D editing algorithm based on Gaussian Splatting (GS), a novel 3D representation technique.

GaussianEditor enhances precision and control in editing through our proposed Gaussian Semantic Tracing, which traces the editing target throughout the training process. Additionally, we propose hierarchical Gaussian splatting (HGS) to achieve stabilized and fine results under stochastic generative guidance from 2D diffusion models. We also develop editing strategies for efficient object removal and integration, a challenging task for existing methods. Our comprehensive experiments demonstrate GaussianEditor's superior control, efficacy, and rapid performance, marking a significant advancement in 3D editing.

Video

Click Image to View Gaussians

Interpolate start reference image.

Interpolate start reference image.
Interpolate start reference image.

Make it Autumn.

Interpolate start reference image.
Interpolate start reference image.

Interpolate start reference image.
Interpolate start reference image.

Remove the tractor.

Interpolate start reference image.
Interpolate start reference image.

Interpolate start reference image.
Interpolate start reference image.

Turn him into an old lady.

Interpolate start reference image.
Interpolate start reference image.

Interpolate start reference image.
Interpolate start reference image.

Turn the bear into a Grizzly bear.

Interpolate start reference image.
Interpolate start reference image.

Interpolate start reference image.
Interpolate start reference image.

Add a dog on the bench.

Interpolate start reference image.
Interpolate start reference image.

Interpolate start reference image.
Interpolate start reference image.

Ablation: Without Semantic Tracing. Turn him into a clown.

Interpolate start reference image.

BibTeX

@misc{chen2023gaussianeditor,
    title={GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting},
    author={Yiwen Chen and Zilong Chen and Chi Zhang and Feng Wang and Xiaofeng Yang and Yikai Wang and Zhongang Cai and Lei Yang and Huaping Liu and Guosheng Lin},
    year={2023},
    eprint={2311.14521},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}