README

Colossal-AI


   

   Colossal-AI: Making large AI models cheaper, faster, and more accessible

     Paper  |
    Documentation  |
    Examples  |
    Forum  |
   GPU Cloud Playground  |
    Blog 

   
   
   
   
   
   
   


   | English | 中文 |



 Instantly Run Colossal-AI on Enterprise-Grade GPUs

Skip the setup. Access a powerful, pre-configured Colossal-AI environment on HPC-AI Cloud.

Train your models and scale your AI workload in one click!

 NVIDIA Blackwell B200s: Experience the next generation of AI performance (See Benchmarks). Now available on cloud from $2.47/hr.
 Cost-Effective H200 Cluster: Get premier performance with on-demand rental from just $1.99/hr.

Get Started Now & Claim Your Free Credits →


   
   
   


 Instant Access Top Open Models at Half the Cost

Skip the hassle. Access powerful, long-context LLMs seamlessly through HPC-AI Model APIs.

Build your AI agents, chatbots, and RAG applications with HPC-AI Model APIs!

 Latest & Greatest Models: Experience state-of-the-art performance with Kimi 2.5, MiniMax 2.5, and GLM 5.1. Perfect for massive 2M+ context windows and complex coding tasks.

 Unbeatable Pricing: Stop overpaying for API endpoints. Get premier inference speed at up to 50% cheaper than OpenRouter.

Get Started Now & Claim Your $4 Free Credits →


   
   
   


 Colossal-AI Benchmark

To see how these performance gains translate to real-world applications, we conducted a large language model training benchmark using Colossal-AI on Llama-like models. The tests were run on both 8-card and 16-card configurations for 7B and 70B models, respectively.

|              GPU              |  GPUs  | Model Size |    Parallelism    | Batch Size per DP | Seqlen | Throughput | TFLOPS/GPU  | Peak Mem(MiB)  |
| :-----------------------------: | :--------: | :-------------: | :------------------: | :-----------: | :--------------: | :-------------: | :-------------: | :-------------: |
|         H200            |     8     |      7B       |   zero2(dp8)     | 36 |        4096     |       17.13 samp/s     |       534.18     |       119040.02     |
|         H200            |     16     |      70B       |   zero2     | 48 |        4096     |       3.27 samp/s     |       469.1     |       150032.23     |
|         B200            |     8     |      7B       |   zero1(dp2)+tp2+pp4     | 128 |        4096     |       25.83 samp/s     |       805.69     |       100119.77     |
|         H200            |     16     |      70B       |   zero1(dp2)+tp2+pp4     | 128 |        4096     |       5.66 samp/s     |       811.79     |       100072.02     |

The results from the Colossal-AI benchmark provide the most practical insight. For the 7B model on 8 cards, the B200 achieved a 50% higher throughput and a significant increase in TFLOPS per GPU. For the 70B model on 16 cards, the B200 again demonstrated a clear advantage, with over 70% higher throughput and TFLOPS per GPU. These numbers show that the B200's performance gains translat

Top issues

Discussions

Latest release

What's Changed
 [HotFix] update load lora model Readme; by @duanjunwen in https://github.com/hpcaitech/ColossalAI/pull/6240
 Update README.md by @Yanjia0 in https://github.com/hpcaitech/ColossalAI/pull/6268
 [ci] update ci by @flybird11111 in https://github.com/hpcaitech/ColossalAI/pull/6254
 [upgrade]Upgrade transformers by @flybird11111 in https://github.com/hpcaitech/ColossalAI/pull/6320
 [release] release version by @flybird11111 in https://github.com/hpcaitech/ColossalAI/pull/6330


Full Changelog: https://github.com/hpcaitech/ColossalAI/compare/v0.4.9...v0.5.0