Skip to content

Releases: daulet/tokenizers

v0.1.1

01 Apr 07:02
Compare
Choose a tag to compare

Move header to root to make it module vendorable. Apparently go tooling ignores non-Go files in subdirectories.

v0.1.0

01 Apr 05:21
Compare
Choose a tag to compare

Initial release with:

  • Load tokenizer from file
  • Encode with add_special_tokens support
  • Decode with skip_special_tokens support
  • Vocab size