GigaToken: ~1000x faster Language model tokenization

fwip

What sort of setups do people have that are bounded by the speed of the tokenizer?

maxdo

Interesting : Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers. The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed. Finally, interactions with Python are minimized, and threads have minimal interactions with each other.

sashank_1509

This is really cool, great work!

0xnyn

I had to stare at that chart for a minute just to let the numbers sink in. It's genuinely mind-bending, incredible ship OP

onlyrealcuzzo

This is awesome, but tokenization is typically <0.1% of total inference time. Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!

semiinfinitely

quite excellent software

zerolines

wow, best release all week.

dmezzetti

Very interesting project! Are there benchmarks for the "compatibility mode" or are all the numbers for the Gigatoken API?

anonymousmoos

Quality software here.

swiftcoder

So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?

vmware508

We should just rewrite everything in Rust, especially bloated Python code, and the world would be a better place. ;) Disclosure: I'm a Rust advocate!

luciana1u

engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable

chocrates

Practically I would need to wait for hugging face models to adopt this? My harness tokenizer is just an estimate since the model tokenizes on my api calls?

apollopower

Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep. When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.

zX41ZdbW

This is exactly what we need! Will try: https://github.com/ClickHouse/ClickHouse/issues/108247 It will be nicer if the README focuses more on per-core performance. About the actual algorithm - will something like matching in a perfect hash table help?

nxpnsv

Surely it should be kilotoken

hansvm

For the lazy among us (not me of course), is there a small number of core techniques which enabled this for even a single architecture and single CPU core?

cschmidt

Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas. And screw all the 0.1% haters on here, this is great stuff.

mcpindex-ai

Tokenization is one of the most under appreciated and under optimized part of the agentic stack- not sure if this is truly production grade, and applicable across all hardware+stack combo but this for sure can help inspire a lot of that work. Good work!

phplovesong

Is tokenizing really the bottleneck? If we go from 20ms to 15ms does it really matter?