[Tokenizers] WhiteSpace pretokenizer not only splitting on whitepace

[luisquintanilla]

Given the following code:

using Microsoft.ML.Tokenizers;

ReadOnlySpan<char> sentence = "[CLS] I love AI [SEP]";

var pretokenizer = new WhiteSpace();

var tokens = pretokenizer.PreTokenize(sentence);

// Actual
Console.WriteLine("Actual");
foreach(var result in tokens)
{
    var substr = sentence.Slice(result.Offset, result.Length);
    Console.WriteLine(substr.ToString());
}

Console.WriteLine("----");

// Expected
Console.WriteLine("Expected");
foreach(var expected in sentence.ToString().Split(' '))
{
    Console.WriteLine(expected);
}

The WhiteSpace tokenizer is splitting based on non-alphanumeric and whitespace characters.

Output:

Actual:
[
CLS
]
I
love
AI
[
SEP
]
----
Expected:
[CLS]
I
love
AI
[SEP]

I would expect it to only split on whitespace based on the name.

[tarekgh]

@luisquintanilla we are doing as what Huggingface doing with the white space pre-tokenizers.

machinelearning/src/Microsoft.ML.Tokenizers/PreTokenizer/Whitespace.cs

Line 22 in 8e3f72d

private const string PretokenizePattern = /*lang=regex*/ @"\w+|[^\w\s]+";

https://github.com/huggingface/tokenizers/blob/fdd26ba9a3f0c133427aab0423888cbde91362d7/tokenizers/src/pre_tokenizers/whitespace.rs#L21

Do you still want to change that?

[dotnet-policy-service]

This issue has been marked needs-author-action and may be missing some important information.