README

AnyText: Multilingual Visual Text Generation And Editing

    

!sample

 📌News
[2025.03.03] - 🔥🔥🔥We have open-sourced AnyText2, which is faster, performs better, and allows you to set properties such as font and color for the text! See [[demo]](https://modelscope.cn/studios/iic/studio_anytext2) [[code]](https://github.com/tyxsspa/AnyText2) [[ckpt]](https://modelscope.cn/models/iic/cv_anytext2)!  
[2024.04.18] - 👏👏👏The training code and dataset(AnyWord-3M) are released!  
[2024.04.18] - You can merge weights from self-trained or community models into AnyText now, including all base models and LoRA models based on SD1.5. Have fun!  
[2024.02.21] - The evaluation code and dataset(AnyText-benchmark) are released.  
[2024.02.06] - Happy Lunar New Year Everyone! We've launched a fun app(表情包大师/MeMeMaster) on ModelScope and HuggingFace to create cute meme stickers. Come and have fun with it!   
[2024.01.17] - 🎉AnyText has been accepted by ICLR 2024(Spotlight)!  
[2024.01.04] - FP16 inference is available, 3x faster! Now the demo can be deployed on GPU with 8GB memory. Enjoy!  
[2024.01.04] - HuggingFace Online demo is available here!  
[2023.12.28] - ModelScope Online demo is available here!  
[2023.12.27] - 🧨We released the latest checkpoint(v1.1) and inference code, check on ModelScope in Chinese.  
[2023.12.05] - The paper is available at here.  

For more AIGC related works of our group, please visit here, and we are seeking collaborators and research interns(Email us).

 ⏰TODOs
- [x] Release the model and inference code
- [x] Provide publicly accessible demo link
- [ ] Provide a free font file(🤔)
- [x] Release tools for merging weights from community models or LoRAs
- [ ] Support AnyText in stable-diffusion-webui(🤔)
- [x] Release AnyText-benchmark dataset and evaluation code
- [x] Release AnyWord-3M dataset and training code
 

 💡Methodology
AnyText comprises a diffusion pipeline with two primary elements: an auxiliary latent module and a text embedding module. The former uses inputs like text glyph, position, and masked image to generate latent features for text generation or editing. The latter employs an OCR model for encoding stroke data as embeddings, which blend with image caption embeddings from the tokenizer to generate texts that seamlessly integrate with the background. We employed text-control diffusion loss and text perceptual loss for training to further enhance writing accuracy.

!framework

 🛠Installation
bash
 Install git (skip if already done)
conda install -c anaconda git
 Clone anytext code
git clone https://github.com/tyxsspa/AnyText.git
cd AnyText
 Prepare a font file; Arial Unicode MS is recommended, you need to download it on your own
mv your/path/to/arialuni.ttf ./font/Arial_Unicode.ttf
 Create a new environment and install packages as follows:
conda env create -f environment.yaml
conda activate anytext


 🔮Inference
[Recommend]： We release a demo on ModelScope and HuggingFace! You can also try AnyText through our API servic