README

aka.ms/GeneralAI
 Hiring
We are hiring at all levels (including FTE researchers and interns)! If you are interested in working with us on Foundation Models (aka large-scale pre-trained models) and General AI, NLP, MT, Speech, Document AI and Multimodal AI, please send your resume to [email protected].

 Foundation Architecture
 TorchScale - A Library of Foundation Architectures (repo)

Fundamental research to develop new architectures for foundation models and AI, focusing on modeling generality and capability, as well as training stability and efficiency.

 Stability - DeepNet: scaling Transformers to 1,000 Layers and beyond

 Generality - Foundation Transformers (Magneto): towards true general-purpose modeling across tasks and modalities (including language, vision, speech, and multimodal)

 Capability - A Length-Extrapolatable Transformer

 Efficiency & Transferability - X-MoE: scalable & finetunable sparse Mixture-of-Experts (MoE)

 The Revolution of Model Architecture

 BitNet: 1-bit Transformers for Large Language Models

 RetNet: Retentive Network: A Successor to Transformer for Large Language Models

 LongNet: Scaling Transformers to 1,000,000,000 Tokens

 Foundation Models

 The Evolution of (M)LLM (Multimodal LLM)

 Kosmos-2.5: A Multimodal Literate Model

 Kosmos-2: Grounding Multimodal Large Language Models to the World

 Kosmos-1: A Multimodal Large Language Model (MLLM)

 MetaLM: Language Models are General-Purpose Interfaces

The Big Convergence - Large-scale self-supervised pre-training across tasks (predictive and generative), languages (100+ languages), and modalities (language, image, audio, layout/format + language, vision + language, audio + language, etc.)



 Language & Multilingual
 UniLM: unified pre-training for language understanding and generation

 InfoXLM/XLM-E: multilingual/cross-lingual pre-trained models for 100+ languages

 DeltaLM/mT6: encoder-decoder pre-training for language generation and translation for 100+ languages

 MiniLM: small and fast pre-trained models for language understanding and generation

 AdaLM: domain, language, and task adaptation of pre-trained models

 EdgeLM(NEW): small pre-trained models on edge/client devices

 SimLM (NEW): large-scale pre-training for similarity matching

 E5 (NEW): text embeddings

 MiniLLM (NEW): Knowledge Distillation of Large Language Models

 Vision
 BEiT/BEiT-2: generative self-supervised pre-training for vision / BERT Pre-Training of Image Transformers

 DiT: self-supervised pre-training for Document Image Transformers

 TextDiffuser/TextDiffuser-2 (NEW): Diffusion Models as Text Painters

 Speech
 WavLM: speech pre-training for full stack tasks

 VALL-E: a neural codec language model for TTS

 Multimodal (X + Language)
 LayoutLM/LayoutLMv2/LayoutLMv3: multimodal (text + layout/format + image) Document Foundation Model for Document AI (e.g. scanned documents, PDF, etc.)

 LayoutXLM: multimodal (text + layout/format + image) Document Foundation Model for multi