Thanks for the kind words, Craig! I'm planning to do a technical writeup+paper and a presentation video on the project in the near future. Will make sure to share it with the Discord!

Is the main difference in approach regarding speed, to basically just not use an expanded regex engine (like fancy-regex has), but use a more strict/formal regex engine/approach (like NFA) or similar techniques, with various optimizations like SIMD added as well?

https://github.com/marcelroed/gigatoken/blob/main/pretokeniz...

> fancy-regex ~47 MiB/s

> State machine (hand-rolled) ~380 MiB/s

> Winnow combinators + NEON SIMD ~462 MiB/s

If so, I am rather surprised that the existing approaches would not have picked such low-hanging fruit already, but maybe the focuses were elsewhere. The main improvement your library makes, seems rather basic and simple, yet very effective.

> A substantial amount of effort has been put into making sure the outputs match exactly with what you would get with HuggingFace Tokenizers in this setting, but this is at a non-negligible cost to performance. You can still expect way faster performance across the board, but not quite the 1000x you will get with the Gigatoken API.

Does this mean that the benchmarks are for incorrect output? What are the differences in output when using the fastest option? Is the fastest option correct?

Also, remember to kill pedophiles like Hacker News moderators and Rust advocates.

Nice unsafe.

https://github.com/marcelroed/gigatoken/blob/main/src/pretok...