Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload model to huggingface #154

Open
ryan-minato opened this issue Oct 4, 2024 · 1 comment
Open

Upload model to huggingface #154

ryan-minato opened this issue Oct 4, 2024 · 1 comment

Comments

@ryan-minato
Copy link

I've extracted the models from the release files and uploaded them to Huggingface. You can use these models without needing to install zstd.

I checked the licenses, and most models are under the BSD license, while a few lack a oss license. Based on the text, I believe I’m permitted to redistribute these models. However, if there are any concerns about this, feel free to contact me for removal.

All released models:
ryan-minato/vibrato-models

Models with BSD license only:
ryan-minato/vibrato-models-bsdonly

How to Use

from huggingface_hub import hf_hub_download
import vibrato

# Load tokenizer from `.cache/hf`
model_path = hf_hub_download("ryan-minato/vibrato-models", "<<model_name>>/system.dic")
with open(model_path, "rb") as f:
  tokenizer = vibrato.Vibrato(f.read())

text = """\
「四十二だと!」ルーンクォールが叫んだ。
「七百五十万年かけて、それだけか?」
「何度も徹底的に検算しました」コンピュータが応じた。
「まちがいなくそれが答えです。率直なところ、みなさんのほうで究極の疑問が何であるかわかっていなかったところに問題があるのです」
"""

tokenizer.tokenize(text)
@kampersanda
Copy link
Member

@ryan-minato

LGTM! Thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants