Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas.

And screw all the 0.1% haters on here, this is great stuff.

▲

marcelroed an hour ago | parent | next [-]

Thanks for the kind words, Craig! I'm planning to do a technical writeup+paper and a presentation video on the project in the near future. Will make sure to share it with the Discord!

	▲	njjkkjjdhs 4 minutes ago \| parent [-]
		Nice unsafe. https://github.com/marcelroed/gigatoken/blob/main/src/pretok...

▲

gghfez 17 minutes ago | parent | prev | next [-]

>the tokenization discord

Could I join this?

▲

cs702 2 hours ago | parent | prev | next [-]

That is my reaction too. It looks like great work!

Valuable not only for inference, but for training too (think proprietary datasets).

I would add, a single individual did this.

One person can make a difference :-)

▲

victor106 an hour ago | parent | prev | next [-]

> tokenization discord

How can I join this? Sounds interesting

	▲	glenngillen 33 minutes ago \| parent [-]
		also subscribing to this!

▲

asdf88990 2 hours ago | parent | prev [-]

[flagged]

▲

cschmidt 2 hours ago | parent [-]

I’m not sure why you think this is ai slop. I work on tokenization research full time. My name is Craig Schmidt and I have a number of papers in the field. This researcher has done some very impressive work and I’m trying to defend him from the HN dismissive hoards.

There is a serious research community on tokenization, and we are quite interested in this work.

	▲	asdf88990 2 hours ago \| parent [-]
		Which part of “doesn’t seem to be aislop signs” made you think I believe your comment was aislop? As I said, It just might be the low information density approach to talking recently that seems off to me. To explain a bit more, I kept reading and waiting for the penny drop but nothing. “I cloned your repo” — okay then what happened? Nothing? I put my shoes on this morning. “I know the tokrniziation community wants to absord tje lessons” okay? Tell me the point please! I know the AI community wants to understand the universe. “These are useful ideas”. Yes. Of course, you could argue that my comment falls in the same category in the sense that I am not actually contributing to the topic at hand but I am peeved with all low the information noise.