Tokenizers cpp 1251

[gabe-l-hart]

Dependencies

• This PR depends on the python-layer support for Granite Code in Granite code support #1336

Description

This PR introduces a collection of new tokenizer capabilities at the C++ level. It is minimally complete at this point, so I'm opening up this PR as a point of discussion. The main goal is to add compatibility with the tokenizers library at the C++ level.

Additions

Dependencies

• Add a third-party dependency on nlohmann/json to support parsing tokenizer.json files from HF tokenizers
• Add a third-party dependency on gtest for unit tests
• Add a vendored copy of the unicode.[h|cpp] and unicode-data.[h|cpp] from llama.cpp. These implement the full set of transformations needed to go to/from the byte-encoded space needed by the ByteLevel functionality.
  • The files are under MIT license, so I copied the license text into a header and left a reference to the commit I used to copy them
  • There is a possible inefficiency when interfacing with these methods since they use const std::string& rather than a string_view (re2::StringPiece), so we need to copy from the view to a string before calling the llama.cpp functions. This could likely be optimized by changing the function signatures in the vendored code, but I tried to avoid any changes to them at this point.
  • I chose to put these directly into the source rather than add them to third_party since this was a slim subset of the full llama.cpp project.

Code Changes

• Introduce the concept of a PreTokenizer and TokenDecoder aligned with the corresponding concepts in the Rust codebase
  • These are not 1:1 ports, but rather a logical port to fit with the existing tokenizer class structure
• Split up Tiktoken into a base implementation of BPE (BPETokenizerBase) with hooks for the specifics of the Tiktoken models
• Add the HFTokenizer as a second derived BPETokenizerBase implementation which adds a PreTokenizer / TokenDecoder to the input/output logic
• Added the tokenize_tool.cpp to sanity check tokenizers. This was mostly just for my own sanity, so it may not have much utility in the future, but I figured it may also be a useful development tool so left it in. It might make sense to move it somewhere else?

Build and Test

• Add the tokenizer/tests dir to hold c++ unit tests
• Add a common main.cpp for all gtest unit tests
• Wire up testing in tokenizer/CMakeLists.txt so that it runs with ctest
  • The test binaries are placed in bin/test and can be executed directly as well
• Added reasonably complete unit tests for the pre_tokenizer stack, but nothing else (yet!)

Testing

Build setup

# Set up the build dir
cd /path/to/torchchat
mkdir -p build/tokenizer
cd build/tokenizer

# Configure cmake to build the unit tests and the tokenize tool
cmake ../../tokenizer -D BUILD_UNIT_TESTS=ON -D BUILD_TOKENIZE_TOOL=ON

# Build it
make -j

Unit testing

# (From build/tokenizer)

# Run all unit tests (currently just the one)
ctest

# Run the test binary directly
./bin/test/pre_tokenizer_test

Spot testing

I also did spot testing with several tokenizer models:

• Granite Code: https://huggingface.co/ibm-granite/granite-3b-code-instruct-128k/blob/main/tokenizer.json
• Llama 3.1 (tiktoken): https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/original/tokenizer.model

# Test with an HF tokenizer json directly
./bin/tokenize_tool hf_tokenizers $HOME/models/ibm-granite/granite-3b-code-instruct-128k/tokenizer.json This is a test 1234 5 foobar

# Test with an HF model repo that has tokenizer.json and tokenizer_config.json
./bin/tokenize_tool hf_tokenizers $HOME/models/ibm-granite/granite-3b-code-instruct-128k This is a test 1234 5 foobar

# Test with a tiktoken tokenizer
./bin/tokenize_tool tiktoken $HOME/models/meta-llama/Meta-Llama-3.1-8B-Instruct/tokenizer.model This is a test 1234 5 foobar

Stil TODO

• Add unit tests for the decoder suite
• Add unit tests for the tokenizers themselves
• Ensure that special tokens are handled correctly with the HFTokenizer (no more reliance on tiktoken hard-coded special tokens)

Future work

• There are still a lot of PreTokenizers and TokenDecoders that are not yet supported. There's also the whole PostProcessors suite as well. A good test case for this would be the tokenizers version of the llama3.1 tokenizer (here)

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1379

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [DomainsOnly] Jobs fail with GLIBC version not found

This comment was automatically generated by Dr. CI and updates every 15 minutes.