Is the main difference in approach regarding speed, to basically just not use an expanded regex engine (like fancy-regex has), but use a more strict/formal regex engine/approach (like NFA) or similar techniques, with various optimizations like SIMD added as well?

https://github.com/marcelroed/gigatoken/blob/main/pretokeniz...

> fancy-regex ~47 MiB/s

> State machine (hand-rolled) ~380 MiB/s

> Winnow combinators + NEON SIMD ~462 MiB/s

If so, I am rather surprised that the existing approaches would not have picked such low-hanging fruit already, but maybe the focuses were elsewhere. The main improvement your library makes, seems rather basic and simple, yet very effective.