Project Group

Goals:

The objective is to develop a pipeline that enhances 3D scene reconstruction by filling in missing data or holes in the 3D model generated by 3D Gaussian Splatting. By leveraging generative AI, the pipeline aims to create realistic in-paintings of incomplete areas, ensuring a more accurate and visually complete 3D representation.

*An example from NeRFFiller of masked area being in-painted in 3D

Approach:

 
- 3DGS Model Training: Begin by training a 3DGS model on the available scene data, generating an initial 3D model.
- Identify Missing Data: Use techniques such as surface normal analysis, depth map inconsistencies, or voxel grid-based methods to detect gaps or holes in the 3D model.
- The problem can be approached in 2 ways, a 2D in-painting and retraining/extension of 3DGS model OR a direct 3D in-painting of splats within the model.
- Prepare Training Data: Extract adjacent frames around the identified incomplete regions. These frames provide diverse views of the missing areas.
- Generative AI Training: Use these frames to train a generative AI model, such as a GAN or a diffusion model, tailored to in-paint and generate complete views from the partial data.
- Model Extension: Apply the generated views to the 3DGS model, extending it to fill in the gaps and improve the overall completeness of the scene.