Speaking for myself. I’ve been aware of a lot of concepts around training, but stayed away from it until very recently.

A few weeks ago, I formed a hypothesis that I could get a smaller model good at the specific standards I have for my git commit messages, which are hard to get out of most LLMs today.

I spent the last four to five weeks on traditional training. It improved in a lot of areas, but not enough to be usable for me.

Yesterday, the article the author of this post is critical of gave me enough of an aha that maybe the problems I was seeing could be solved by having a larger LLM change the weights directly, without going through training.

In the last twelve-ish hours I’ve made more headway than I have in 2+ weeks. I’m using one of my last well performing checkpoints (full fine-tune + changes to the decision head/LoRA, etc.) and gave the model free rein to change any of the layers.

Its last two actions: (a) removed 16 participant-order directions from a head matrix, which improved one type of reasoning and degraded another, and (b) restored one of the 16, which reversed the degradation while keeping the core improvement.

All that to say, this might be known to people who’ve been around training for years, even decades, but it’s new to a lot of us. And we’re excited about it.

▲

runitx 3 hours ago | parent | next [-]

Would you be willing to share what the specific standards for your git commit messages are, and with what models you used that those standards couldn't be met?

Generally speaking you should be able to do something like that in AGENTS, I am really surprised that you have had an issue with most LLMs today getting them to follow a standard for SDLC. Personally, I like conventional commits a lot.

▲

bushido 3 hours ago | parent [-]

> Generally speaking you should be able to do something like that in AGENTS

I really wish that was true. All agents find some way to break the rules, and leave messages for their future selves.

The agents always find a way to include some forms of hedging, contradictions, future-looking documentation (which doesn't age well), etc. into git messages.

I found having an extra gate (husky/pre-commit hook), even if it's not as good as I want it to be improves the quality of how the AI writes substantially.

The cost of it not being very good is the agents now spend more time validating the claims of the validator. It's actually hilarious to witness.

I have been using local models for this final verification, which is partly why they're not that great. It's also why I decided to train my own.

It's been a great, fun learning experience. I learn almost everything via building. So why change a recipe that works :)

	▲	runitx 3 hours ago \| parent [-]
		I have not had an experience anything like you are describing with Claude. I have found it to be the most reliable product I have ever purchased, actually. Learned it quickly, read the documentation, and it works. Local models are not at all comparable to Claude to me, and I have used them: Qwen, DeepSeek, Minimax, Stepfun, Nemotron. None of them as good as Claude Fabel. If your model is not adhering to AGENTS.md again I am really curious what exact model you are using. I am really just astonished that whatever model you are using is ignoring a clearly defined git conventions standards in your AGENTS.md in 2026.

▲

HappyDreamer 3 hours ago | parent | prev | next [-]

> it’s new to a lot of us. And we’re excited about it.

Yes definitely.

Seems to me as if the weighs fit in the big model's context, it (the big model) reads all weights and doesn't need python.

But if the weights don't fit in context, then it uses python.

Isn't that pretty obvious in a way? That if too many weights, can't look at all. I don't think there's a need for OP to be dismissive when the AI uses python to look at a subset of the weights. It just has to?

To me, Astra understanding a 200k weights small model (the stuck car) is amazing enough already.

▲

syllogism 3 hours ago | parent [-]

No that's not a good summary.

The model can only reason from the weights directly when there's a fairly linear relationship between the behaviour and the weight values. In a large model that's not remotely true.

Using Python to debug and fix problems with the model is totally different. It doesn't read in any of the weight values into the context at all when it uses the Python approach. Rather it gathers diagnostic statistics about the weights. It does just the same sort of thing a machine learning researcher would do to solve the problem.

So there are basically two unrelated types of experiments in the paper. In one type of experiment, the model reads the weights like it reads an image. In fact if you visualise the weights it's fairly clear why this works.

In the other type of experiment, the model is tasked with writing Python code to do ML engineering. This requires writing Python programs to analyse and manipulate weight data. This is a difficult coding problem but it really is just a coding problem. It doesn't "look" at the weights in these tasks any more than you or I would look at the weights.

	▲	HappyDreamer 44 minutes ago \| parent [-]
		> It does just the same sort of thing a machine learning researcher would do to solve the problem. [...] It doesn't "look" at the weights in these tasks any more than you or I would look at the weights. Ok. Thanks for making that more clear! > two unrelated types of experiments ... The first type of experiment (small nets that seem to fit in context) are enough to make me impressed actually :-) But yes then maybe the previous article could have been split in two, or two sections or something.

▲

ghm2180 3 hours ago | parent | prev | next [-]

I assume you run a benchmark on all the tasks on every change the model recommends? Do you have it set up like nanogpt speed run thingy that karpathy for a thumbs up or down for every change when you did this with astra?

▲

dataflow 2 hours ago | parent | prev | next [-]

Wow. Are you saying you trained a smaller model from scratch (from what training data?), then used a larger model (which model?) to tweak its weights to fix this problem? Or did you post-train/fine-tune an existing model? I'm very curious what models, hardware, and training data you used!

	▲	bushido an hour ago \| parent [-]
		> Are you saying you trained a smaller model from scratch Not yet - but I really want to. Current experiments are on Qwen3.5-2B, and I've done a bunch on ModernBERT-large. My main hypothesis is: I'm fairly confident I can come up with a university curriculum for a high school student to learn certain skills so that even without gaining technical knowledge, they can call bullshit on commit messages written by Claude and Codex. And I'm basically converting that curriculum into training. Training a model from scratch is likely the next thing I'll do once I hit a significant bottle neck. There are still quite a few things to try (ex: Franken-merging, converting the dense model into a MoE, etc). > very curious what models, hardware Mostly using run pod and similar, and honestly, grabbing any decently sized GPU that I can get my hands on. 5090 - B200s. > training data Public data and synthetic data. This has been the hardest part. Also, why I'm excited about not having to "train" the behavior into the models.

▲

overfeed 3 hours ago | parent | prev | next [-]

From the Fine Article you're calling critical:

>> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights

▲

syllogism 3 hours ago | parent | prev [-]

Out of interest, did you use an assistant like Claude Code or Codex to help with the training experiments?