Skip to content

Commit

Permalink
Merge pull request #813 from EleutherAI/fix-tok-import
Browse files Browse the repository at this point in the history
fix unexpected indent error for conditional `tiktoken` import
  • Loading branch information
StellaAthena authored Mar 8, 2023
2 parents a5c2229 + 019da08 commit 71df4d5
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 6 deletions.
2 changes: 1 addition & 1 deletion configs/neox_arguments.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Logging Arguments

- **git_hash**: str

Default = 496f987
Default = 5face7a

current git hash of repository

Expand Down
10 changes: 5 additions & 5 deletions megatron/tokenizer/tokenizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -352,11 +352,11 @@ def eod(self):

class TiktokenTokenizer(AbstractTokenizer):
"""Tokenizer from OpenAI's tiktoken implementation"""
try:
import tiktoken
except ModuleNotFoundError:
print("Please install tiktoken: (https://github.com/openai/tiktoken)")
raise Exception
try:
import tiktoken
except ModuleNotFoundError:
print("Please install tiktoken: (https://github.com/openai/tiktoken)")
raise Exception

def __init__(self, vocab_file):
name = "TiktokenTokenizer"
Expand Down

0 comments on commit 71df4d5

Please sign in to comment.