Releases: daulet/tokenizers
Releases · daulet/tokenizers
v0.1.1
Move header to root to make it module vendorable. Apparently go tooling ignores non-Go files in subdirectories.
v0.1.0
Initial release with:
- Load tokenizer from file
- Encode with
add_special_tokens
support
- Decode with
skip_special_tokens
support
- Vocab size