> Are you saying you trained a smaller model from scratch

Not yet - but I really want to. Current experiments are on Qwen3.5-2B, and I've done a bunch on ModernBERT-large.

My main hypothesis is: I'm fairly confident I can come up with a university curriculum for a high school student to learn certain skills so that even without gaining technical knowledge, they can call bullshit on commit messages written by Claude and Codex. And I'm basically converting that curriculum into training.

Training a model from scratch is likely the next thing I'll do once I hit a significant bottle neck. There are still quite a few things to try (ex: Franken-merging, converting the dense model into a MoE, etc).

> very curious what models, hardware

Mostly using run pod and similar, and honestly, grabbing any decently sized GPU that I can get my hands on. 5090 - B200s.

> training data

Public data and synthetic data. This has been the hardest part. Also, why I'm excited about not having to "train" the behavior into the models.