semchunk

semchunk is a fast and lightweight Python library for splitting text into semantically meaningful chunks.

Owing to its complex yet highly efficient chunking algorithm, semchunk is both more semantically accurate than langchain.text_splitter.RecursiveCharacterTextSplitter (see How It Works 🔍) and is also over 80% faster than semantic-text-splitter (see the Benchmarks 📊).

Installation 📦

semchunk may be installed with pip:

pip install semchunk

Usage 👩‍💻

The code snippet below demonstrates how text can be chunked with semchunk:

import semchunk
from transformers import AutoTokenizer # Neither `transformers` nor `tiktoken` are required,
import tiktoken                        # they are here for demonstration purposes.

chunk_size = 2 # A low chunk size is used here for demonstration purposes. Keep in mind that
               # `semchunk` doesn't take special tokens into account unless you're using a
               # custom token counter, so you probably want to deduct your chunk size by the
               # number of special tokens added by your tokenizer.
text = 'The quick brown fox jumps over the lazy dog.'

# As you can see below, `semchunk.chunkerify` will accept the names of all OpenAI models, OpenAI
# `tiktoken` encodings and Hugging Face models (in that order of precedence), along with custom
# tokenizers that have an `encode()` method (such as `tiktoken`, `transformers` and `tokenizers`
# tokenizers) and finally any function that can take a text and return the number of tokens in it.
chunker = semchunk.chunkerify('umarbutler/emubert', chunk_size) or \
          semchunk.chunkerify('gpt-4', chunk_size) or \
          semchunk.chunkerify('cl100k_base', chunk_size) or \
          semchunk.chunkerify(AutoTokenizer.from_pretrained('umarbutler/emubert'), chunk_size) or \
          semchunk.chunkerify(tiktoken.encoding_for_model('gpt-4'), chunk_size) or \
          semchunk.chunkerify(lambda text: len(text.split()), chunk_size)

# The resulting `chunker` can take and chunk a single text or a list of texts, returning a list of
# chunks or a list of lists of chunks, respectively.
assert chunker(text) == ['The quick', 'brown', 'fox', 'jumps', 'over the', 'lazy', 'dog.']
assert chunker([text], progress = True) == [['The quick', 'brown', 'fox', 'jumps', 'over the', 'lazy', 'dog.']]

# If you have a large number of texts to chunk and speed is a concern, you can also enable
# multiprocessing by setting `processes` to a number greater than 1.
assert chunker([text], processes = 2) == [['The quick', 'brown', 'fox', 'jumps', 'over the', 'lazy', 'dog.']]

Chunkerify

def chunkerify(
    tokenizer_or_token_counter: str | tiktoken.Encoding | transformers.PreTrainedTokenizer | \
                                tokenizers.Tokenizer | Callable[[str], int],
    chunk_size: int = None,
    max_token_chars: int = None,
    memoize: bool = True,
) -> Callable[[str | Sequence[str], bool, bool], list[str] | list[list[str]]]:

chunkerify() constructs a chunker that splits one or more texts into semantically meaningful chunks of a specified size as determined by the provided tokenizer or token counter.

tokenizer_or_token_counter is either: the name of a tiktoken or transformers tokenizer (with priority given to the former); a tokenizer that possesses an encode attribute (eg, a tiktoken, transformers or tokenizers tokenizer); or a token counter that returns the number of tokens in a input.

chunk_size is the maximum number of tokens a chunk may contain. It defaults to None in which case it will be set to the same value as the tokenizer's model_max_length attribute (deducted by the number of tokens returned by attempting to tokenize an empty string) if possible otherwise a ValueError will be raised.

max_token_chars is the maximum numbers of characters a token may contain. It is used to significantly speed up the token counting of long inputs. It defaults to None in which case it will either not be used or will, if possible, be set to the numbers of characters in the longest token in the tokenizer's vocabulary as determined by the token_byte_values or get_vocab methods.

memoize flags whether to memoize the token counter. It defaults to True.

This function returns a chunker that takes either a single text or a sequence of texts and returns, if a single text has been provided, a list of chunks up to chunk_size-tokens-long with any whitespace used to split the text removed, or, if multiple texts have been provided, a list of lists of chunks, with each inner list corresponding to the chunks of one of the provided input texts.

The resulting chunker can be passed a processes argument that specifies the number of processes to be used when chunking multiple texts.

It is also possible to pass a progress argument which, if set to True and multiple texts are passed, will display a progress bar.

Technically, the chunker will be an instance of the semchunk.Chunker class to assist with type hinting, though this should have no impact on how it can be used.

Chunk

def chunk(
    text: str,
    chunk_size: int,
    token_counter: Callable,
    memoize: bool = True,
) -> list[str]

chunk() splits a text into semantically meaningful chunks of a specified size as determined by the provided token counter.

text is the text to be chunked.

chunk_size is the maximum number of tokens a chunk may contain.

token_counter is a callable that takes a string and returns the number of tokens in it.

memoize flags whether to memoize the token counter. It defaults to True.

This function returns a list of chunks up to chunk_size-tokens-long, with any whitespace used to split the text removed.

How It Works 🔍

semchunk works by recursively splitting texts until all resulting chunks are equal to or less than a specified chunk size. In particular, it:

1. Splits text using the most semantically meaningful splitter possible;
2. Recursively splits the resulting chunks until a set of chunks equal to or less than the specified chunk size is produced;
3. Merges any chunks that are under the chunk size back together until the chunk size is reached; and
4. Reattaches any non-whitespace splitters back to the ends of chunks barring the final chunk if doing so does not bring chunks over the chunk size, otherwise adds non-whitespace splitters as their own chunks.

To ensure that chunks are as semantically meaningful as possible, semchunk uses the following splitters, in order of precedence:

1. The largest sequence of newlines (\n) and/or carriage returns (\r);
2. The largest sequence of tabs;
3. The largest sequence of whitespace characters (as defined by regex's \s character class);
4. Sentence terminators (., ?, ! and *);
5. Clause separators (;, ,, (, ), [, ], “, ”, ‘, ’, ', " and `);
6. Sentence interrupters (:, — and …);
7. Word joiners (/, \, –, & and -); and
8. All other characters.

Benchmarks 📊

On a desktop with a Ryzen 9 7900X, 96 GB of DDR5 5600MHz CL40 RAM, Windows 11 and Python 3.12.4, it takes semchunk 2.87 seconds to split every sample in NLTK's Gutenberg Corpus into 512-token-long chunks with GPT-4's tokenizer (for context, the Corpus contains 18 texts and 3,001,260 tokens). By comparison, it takes semantic-text-splitter (with multiprocessing) 25.03 seconds to chunk the same texts into 512-token-long chunks — a difference of 88.53%.

The code used to benchmark semchunk and semantic-text-splitter is available here.

Licence 📄

This library is licensed under the MIT License.