README

🦾 OpenLLM: Self-Hosting LLMs Made Easy









OpenLLM allows developers to run any open-source LLMs (Llama 3.3, Qwen2.5, Phi3 and more) or custom models as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Docker and Kubernetes.

Understand the design philosophy of OpenLLM.

 Get Started

Run the following commands to install OpenLLM and explore it interactively.

bash
pip install openllm   or pip3 install openllm
openllm hello


!hello

 Supported models

OpenLLM supports a wide range of state-of-the-art open-source LLMs. You can also add a model repository to run custom models with OpenLLM.


  
    Model
    Parameters
    Required GPU
    Start a Server
  
  
    deepseek
    r1-671b
    80Gx16
    openllm serve deepseek:r1-671b
  
  
    gemma2
    2b
    12G
    openllm serve gemma2:2b
  
  
    gemma3
    3b
    12G
    openllm serve gemma3:3b
  
  
    jamba1.5
    mini-ff0a
    80Gx2
    openllm serve jamba1.5:mini-ff0a
  
  
    llama3.1
    8b
    24G
    openllm serve llama3.1:8b
  
  
    llama3.2
    1b
    24G
    openllm serve llama3.2:1b
  
  
    llama3.3
    70b
    80Gx2
    openllm serve llama3.3:70b
  
  
    llama4
    17b16e
    80Gx8
    openllm serve llama4:17b16e
  
  
    mistral
    8b-2410
    24G
    openllm serve mistral:8b-2410
  
  
    mistral-large
    123b-2407
    80Gx4
    openllm serve mistral-large:123b-2407
  
  
    phi4
    14b
    80G
    openllm serve phi4:14b
  
  
    pixtral
    12b-2409
    80G
    openllm serve pixtral:12b-2409
  
  
    qwen2.5
    7b
    24G
    openllm serve qwen2.5:7b
  
  
    qwen2.5-coder
    3b
    24G
    openllm serve qwen2.5-coder:3b
  
  
    qwq
    32b
    80G
    openllm serve qwq:32b
  


For the full model list, see the OpenLLM models repository.

 Start an LLM server

To start an LLM server locally, use the openllm serve command and specify the model version.

 [!NOTE]
 OpenLLM does not store model weights. A Hugging Face token (HF_TOKEN) is required for gated models.

 1. Create your Hugging Face token here.
 2. Request access to the gated model, such as meta-llama/Llama-3.2-1B-Instruct.
 3. Set your token as an environment variable by running:
    bash
    export HF_TOKEN=
    

bash
openllm serve llama3.2:1b


The server will be accessible at http://localhost:3000, providing OpenAI-compatible APIs for interaction. You can call the endpoints with different frameworks and tools that support OpenAI-compatible APIs. Typically, you may need to specify the following:

- The API host address: By default, the LLM is hosted at http://localhost:3000.
- The model name: The name can be different depending on the tool you use.
- The API key: The API key used for client authentication. This is optional.

Here are some examples:



OpenAI Python client

python
from openai import OpenAI

client = OpenAI(base_url='http://localhost:30