Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Fix: `sentencepiece` tokenizers with added tokens failed with an incorrect assertion
  • Loading branch information
goerch authored Oct 7, 2023
1 parent 1faaae8 commit 3a716b4
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions llama.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8200,7 +8200,9 @@ int llama_token_to_piece(const struct llama_model * model, llama_token token, ch
buf[0] = llama_token_to_byte(model->vocab, token);
return 1;
} else {
GGML_ASSERT(false);
// TODO: for now we accept all unsupported token types,
// suppressing them like CONTROL tokens.
// GGML_ASSERT(false);
}
break;
}
Expand All @@ -8216,7 +8218,9 @@ int llama_token_to_piece(const struct llama_model * model, llama_token token, ch
} else if (llama_is_control_token(model->vocab, token)) {
;
} else {
GGML_ASSERT(false);
// TODO: for now we accept all unsupported token types,
// suppressing them like CONTROL tokens.
// GGML_ASSERT(false);
}
break;
}
Expand Down

0 comments on commit 3a716b4

Please sign in to comment.