README

Moshi: a speech-text foundation model for real time dialogue

!precommit badge
!rust ci badge

[[Read the paper]][moshi] [[Demo]](https://moshi.chat) [[Hugging Face]](https://huggingface.co/collections/kyutai/moshi-v01-release-66eaeaf3302bef6bd9ad7acd)

[Moshi][moshi] is a speech-text foundation model and full-duplex spoken dialogue framework.
It uses [Mimi][moshi], a state-of-the-art streaming neural audio codec.
Talk to Moshi now in our live demo.

 Organisation of the repository

There are three separate versions of the Moshi inference stack in this repo.


- PyTorch: for research and tinkering. The code is in the moshi/ directory.
- MLX: for on-device inference on iPhone and Mac. The code is in the moshi_mlx/ directory.
- Rust: for production. The code is in the rust/ directory.
    This contains in particular a Mimi implementation in Rust, with Python bindings available
    as rustymimi.

Finally, the code for the web UI client used in the Moshi demo is provided in the client/ directory.

If you want to fine tune Moshi, head out to kyutai-labs/moshi-finetune.

 Other Kyutai models

The Moshi codebase is also used to run related models from Kyutai that use a multi-stream architecture similar to Moshi:
- Hibiki: simultaneous speech translation. Check out the Hibiki repo for more info.
- Kyutai Text-To-Speech and Speech-To-Text. Check out the Delayed Streams Modeling repo for more info.

 Model architecture

Moshi models two streams of audio: one corresponds to Moshi speaking, and the other one to the user speaking.
Along with these two audio streams, Moshi predicts text tokens corresponding to its own speech, its inner monologue,
which greatly improves the quality of its generation.
A small Depth Transformer models inter-codebook dependencies for a given time step,
while a large, 7B-parameter Temporal Transformer models the temporal dependencies. Moshi achieves a theoretical latency
of 160ms (80ms for the frame size of Mimi + 80ms of acoustic delay), with a practical overall latency as low as 200ms on an L4 GPU.




 Mimi

Mimi is a neural audio codec that processes 24 kHz audio, down to a 12.5 Hz representation
with a bandwidth of 1.1 kbps, in a fully streaming manner (latency of 80ms, the frame size),
yet performs better than existing, non-streaming, codecs like
SpeechTokenizer (50 Hz, 4kbps), or SemantiCodec (50 Hz, 1.3kbps).

Mimi builds on previous neural audio codecs such as SoundStream
and EnCodec, adding a Transformer both in the encoder and decoder,
and adapting the strides to match an overall frame rate of 12.5 Hz. This allows Mimi to get closer to the
average frame rate of text tokens (~3-4 Hz), and limit the number of autoregressive steps in Moshi.
Similarly to SpeechTokenizer, Mimi uses a distillation loss so that the first codebook tokens match
a self-supervised representation from WavLM, which allows modeling semantic and acoustic information with a single model. Finally, and similarly to EBEN,
Mimi uses only an adversarial trai