This tracks with my experience. A core and integral part to make models actually shine involves post training and custom harness engineering, all specifically done for the purpose of getting them to settle into competent inputs and outputs that are relevant to you, optimized for the harness you build which better suits your domain. The weights are just a generalization. A block of marble to be sculpted.

As the compute to build adaptions for multi-billion to trillion parameter models becomes more and more available (and affordable), and the artistic techniques of fine tuning and harness engineering spread throughout the public, I think we'll come to see the "one size fits all" model for the non-starter it really is. Anybody who's even toyed around with LoRAs (let alone made their own) already knows this. That's a very deep rabbit hole, and the ceiling is determined by target domain knowledge and systems engineering.

I'm bullish on LLMs as expert tools in the same grain that computers are. You have to learn all about them to use them effectively. But what really makes the difference is how well you know what you're going to be pointing them at. There is very little room for companies like OpenAI or Google to live between us and our tools.

▲

anonymousDan an hour ago | parent [-]

Sorry not an AI specialist, what exactly do you mean by 'custom harness engineering'? Some way of informing the model whether the output it is producing is good or bad based on the specific task in question?

	▲	ux266478 30 minutes ago \| parent [-]
		Harnesses are the control surface the model interacts with. How it invokes tools, the tools it has access to, how agents are coordinated. It's like an interface or a shell. It's the magic that lets an LLM operate a computer. You can read more here: https://learn.microsoft.com/en-us/agent-framework/concepts/h... > Some way of informing the model whether the output it is producing is good or bad That's what post-training is for. At its most basic, you're giving it examples of inputs and outputs and then doing reinforcement learning to calibrate its adaptation to your examples. You need much less to fine tune a model than you do to pretrain the weights. You can build a really excellent LoRA for a stable diffusion model, for example, with 50 high quality images. LLMs are "a bit" more complicated and costly to fine tune, and you have to be mindful of the agentic loop, but the principle is the same. There's more to it than just LoRAs. Steering vectors, projection layers, custom encoders, etc. There's a fair amount to learn, but it sounds a lot scarier than it is. Here's something to chew on: chain-of-thought doesn't exist until after pretraining! It's basically created by having <think>...</think> blocks directly in some example outputs, and this is fine-tuned into stability. It's literally not much more than a parlor trick and some careful calibration. A powerful parlor trick to be sure, though.