openai /tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

★ 19,626⑂ 1,673
PythonCreated 2022-12-01
19,626
stars
1,673
forks
–
watchers
–
prs
–
issues
–
releases
–
discussions

Snapshot: 2026-11-10T09:35:25Z · Data license Apache-2.0 · JSON