Agree with this - started programming through learning scripting in ROBLOX when I was like 12 (this was back like 17 or 18 years ago) and it developed into a life-long passion for software engineering. I am thankful I had people around me (my parents), who were aware enough to realize I wasn't just playing video games and gave me the time I needed on the computer to learn and experiment with programming...

This also meant that by the time I was actually offered to take a programming class in school (junior year of HS), I had already been able to self-teach myself well beyond what that class was covering, thanks to just working on random projects that scratched an itch I had at the time, looking up anything I didn't know or understand, and internalizing those concepts over time.

In short though, I definitely agree, young kids and teens (and also, frankly, adults too!) should be encouraged to explore things that they have a passion for, without being told 'you need to go to school for this' or 'you cant understand this at your age'

I just voiced my opinion. I just think buiding an LLM from the scratch for 17 y.o. is pointless exercise, advising a teenager to do so is borderline irresponsible, and frankly PG is simply virtue signalling here, as LLMs are still trendy, esp. in his circles.

There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IMO_ waste of of time, with illusion of understanding w/o actually having any.

> Why would you tell people that the correct order

Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17?

> For some (many?) people, a 'proper' understanding develops _after_ the exploration.

I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be unable to build LLM drom scratch, unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring".

If I were in "exploratory" state of mins, I'd rather go from entirely different side - I'd try playing with LoRA-ing existing small LLMs, such as venerable 2 y.o. Mistral Nemo, to get "feeling" for what training is and how hyperameters influence the process.

That'd would be a terrible advice if there weren't a plenty of other things "you are not equipped for", but far less daunting both theoretically and practically. Such as, say, convolutional neural networks, or some older ML tech. Or even something totally unrelated to ML.

Transformers are difficult to understand even to people with strong ML background, let alone a teenager.

How would you filter out garbage from your training data, for example? If you are trying to use someone elses corpus, would it be "from the scratch" then?