README

Omost

Omost is a project to convert LLM's coding capability to image generation (or more accurately, image composing) capability. 

The name Omost (pronunciation: almost) has two meanings: 1) everytime after you use Omost, your image is almost there; 2) the O mean "omni" (multi-modal) and most means we want to get the most out of it.

Omost provides LLMs models that will write codes to compose image visual contents with Omost's virtual Canvas agent. This Canvas can be rendered by specific implementations of image generators to actually generate images.

Currently, we provide 3 pretrained LLM models based on variations of Llama3 and Phi3 (see also the model notes at the end of this page).

All models are trained with mixed data of (1) ground-truth annotations of several datasets including Open-Images, (2) extracted data by automatically annotating images, (3) reinforcement from DPO (Direct Preference Optimization, "whether the codes can be compiled by python 3.10 or not" as a direct preference), and (4) a small amount of tuning data from OpenAI GPT4o's multi-modal capability.

 Get Started

You can just use the official HuggingFace space.

Or, you can use the below deployment (requires 8GB Nvidia VRAM):

    git clone https://github.com/lllyasviel/Omost.git
    cd Omost
    conda create -n omost python=3.10
    conda activate omost
    pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
    pip install -r requirements.txt
    python gradio_app.py

(Note that quant LLM requires bitsandbytes - some 9XX or 10XX or 20XX GPUs may have trouble in running it. If that happens, just use our official huggingface space.)

 Screenshots

(All with random seed 12345)

a ragged man wearing a tattered jacket in the nineteenth century：

!image

!image

the greatest escape after the apocalypse, the hope of all humanity：

!image

!image

jurassic dinosaur battle：

!image

!image

the supreme ruler with endless magic on a gothic throne, with undead armies kneeling:

!image

!image

a funny cartoon batman fights joker:

!image

!image

 Example Transcripts (with Conversational Editing)

(Below is a raw transcript from default model, with random seed 123456)

(If the GitHub's code block rendering is causing troubles to reading long lines, click here to read raw texts.)

 User

generate an image of the fierce battle of warriors and the dragon

 Assistant

python
 Initialize the canvas
canvas = Canvas()

 Set a global description for the canvas
canvas.set_global_description(
    description='A fierce battle between warriors and a dragon.',
    detailed_descriptions=[
        'In this intense scene, a group of fierce warriors is engaged in an epic battle with a mighty dragon.',
        'The warriors, clad in armor and wielding swords and shields, are positioned on the left side of the image.',
        'Their expressions are determined and focused, reflecting their resolve to defeat the dragon.',
        'The dragon, with its massive wings spr