tiktoken 0.12.0_4
textproc/py-tiktoken
Fast BPE tokeniser for use with OpenAI's models
Description
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Dependencies
- build devel/py-build
- build devel/py-installer
- build devel/py-setuptools
- build devel/py-setuptools-rust
- build devel/py-wheel
- build lang/python311
- build lang/rust
- run lang/python311
- run textproc/py-regex
- run www/py-requests
Commit History
may be incomplete — full history at freebsd-ports on GitHub
Loading commit history — this may take up to a minute on first view. Reload the page in a moment.