README

LlamaGPT
  
    A self-hosted, offline, ChatGPT-like chatbot, powered by Llama 2. 100% private, with no data leaving your device.
    
    New: Support for Code Llama models and Nvidia GPUs.
    
    
    umbrel.com (we're hiring) »
    
    
    
      
    
    
      
    
    
      
    
    
      
    
  


  
    
  


 Contents

1. Demo
2. Supported Models
3. How to install
   - On umbrelOS home server
   - On M1/M2 Mac
   - Anywhere else with Docker
   - Kubernetes
4. OpenAI-compatible API
5. Benchmarks
6. Roadmap and contributing
7. Acknowledgements

 Demo

https://github.com/getumbrel/llama-gpt/assets/10330103/5d1a76b8-ed03-4a51-90bd-12ebfaf1e6cd

 Supported models

Currently, LlamaGPT supports the following models. Support for running custom models is on the roadmap.

| Model name                               | Model size | Model download size | Memory required |
| ---------------------------------------- | ---------- | ------------------- | --------------- |
| Nous Hermes Llama 2 7B Chat (GGML q4_0)  | 7B         | 3.79GB              | 6.29GB          |
| Nous Hermes Llama 2 13B Chat (GGML q4_0) | 13B        | 7.32GB              | 9.82GB          |
| Nous Hermes Llama 2 70B Chat (GGML q4_0) | 70B        | 38.87GB             | 41.37GB         |
| Code Llama 7B Chat (GGUF Q4_K_M)         | 7B         | 4.24GB              | 6.74GB          |
| Code Llama 13B Chat (GGUF Q4_K_M)        | 13B        | 8.06GB              | 10.56GB         |
| Phind Code Llama 34B Chat (GGUF Q4_K_M)  | 34B        | 20.22GB             | 22.72GB         |

 How to install

 Install LlamaGPT on your umbrelOS home server

Running LlamaGPT on an umbrelOS home server is one click. Simply install it from the Umbrel App Store.



 Install LlamaGPT on M1/M2 Mac

Make sure your have Docker and Xcode installed.

Then, clone this repo and cd into it:


git clone https://github.com/getumbrel/llama-gpt.git
cd llama-gpt


Run LlamaGPT with the following command:


./run-mac.sh --model 7b


You can access LlamaGPT at http://localhost:3000.

 To run 13B or 70B chat models, replace 7b with 13b or 70b respectively.
 To run 7B, 13B or 34B Code Llama models, replace 7b with code-7b, code-13b or code-34b respectively.

To stop LlamaGPT, do Ctrl + C in Terminal.

 Install LlamaGPT anywhere else with Docker

You can run LlamaGPT on any x86 or arm64 system. Make sure you have Docker installed.

Then, clone this repo and cd into it:


git clone https://github.com/getumbrel/llama-gpt.git
cd llama-gpt


Run LlamaGPT with the following command:


./run.sh --model 7b


Or if you have an Nvidia GPU, you can run LlamaGPT with CUDA support using the --with-cuda flag, like:


./run.sh --model 7b --with-cuda


You can access LlamaGPT at http://localhost:3000.

 To run 13B or 70B chat models, replace 7b with 13b or 70b respectively.
 To run Code Llama 7B, 13B or 34B models, replace 7b with code-7b, code-13b or code-34b respectively.

To stop LlamaGPT, do Ctrl + C in Terminal.

 Note: O