README

Stable-Dreamfusion

A pytorch implementation of the text-to-3D model Dreamfusion, powered by the Stable Diffusion text-to-2D model.

ADVERTISEMENT: Please check out threestudio for recent improvements and better implementation in 3D content generation!

NEWS (2023.6.12):

 Support of Perp-Neg to alleviate multi-head problem in Text-to-3D.
 Support of Perp-Neg for both Stable Diffusion and DeepFloyd-IF.

https://user-images.githubusercontent.com/25863658/236712982-9f93bd32-83bf-423a-bb7c-f73df7ece2e3.mp4

https://user-images.githubusercontent.com/25863658/232403162-51b69000-a242-4b8c-9cd9-4242b09863fa.mp4

 Update Logs

 Colab notebooks:
 Instant-NGP backbone (-O): 

 Vanilla NeRF backbone (-O2): 

 Important Notice
This project is a work-in-progress, and contains lots of differences from the paper. The current generation quality cannot match the results from the original paper, and many prompts still fail badly!

 Notable differences from the paper
 Since the Imagen model is not publicly available, we use Stable Diffusion to replace it (implementation from diffusers). Different from Imagen, Stable-Diffusion is a latent diffusion model, which diffuses in a latent space instead of the original image space. Therefore, we need the loss to propagate back from the VAE's encoder part too, which introduces extra time cost in training.
 We use the multi-resolution grid encoder to implement the NeRF backbone (implementation from torch-ngp), which enables much faster rendering (~10FPS at 800x800).
 We use the Adan optimizer as default.

 Install

bash
git clone https://github.com/ashawkey/stable-dreamfusion.git
cd stable-dreamfusion


 Optional: create a python virtual environment

To avoid python package conflicts, we recommend using a virtual environment, e.g.: using conda or venv:

bash
python -m venv venv_stable-dreamfusion
source venv_stable-dreamfusion/bin/activate  you need to repeat this step for every new terminal


 Install with pip

bash
pip install -r requirements.txt


 Download pre-trained models

To use image-conditioned 3D generation, you need to download some pretrained checkpoints manually:
 Zero-1-to-3 for diffusion backend.
    We use zero123-xl.ckpt by default, and it is hard-coded in guidance/zero123_utils.py.
    bash
    cd pretrained/zero123
    wget https://zero123.cs.columbia.edu/assets/zero123-xl.ckpt
    
 Omnidata for depth and normal prediction.
    These ckpts are hardcoded in preprocess_image.py.
    bash
    mkdir pretrained/omnidata
    cd pretrained/omnidata
     assume gdown is installed
    gdown '1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI&confirm=t'  omnidata_dpt_depth_v2.ckpt
    gdown '1wNxVO4vVbDEMEpnAi_jwQObf2MFodcBR&confirm=t'  omnidata_dpt_normal_v2.ckpt
    

To use DeepFloyd-IF, you need to accept the usage conditions from hugging face, and login with huggingface-cli login in command line.

For DMTet, we port the pre-generated 32/64/128 resolution tetrahedron grids under tets.
The 256 resolution one can be found here.