Do you then recommend Sol for technical documentation?

What's confusing to me in addition to the naming is that there are no published guidelines I can use, as far as I can see. Put yourself in the user's shoes. On what basis should one pick a model for a given task?

I can understand smaller models being dumber and faster. That aligns well with scenarios where you are willing to compromise; e.g. you're classifying content or asking a model to format text into a table.

But when you want "maximum intelligence, cost and time be damned", there's the implication that bigger models are smarter on a linear scale, and every provider's model release naming is currently conflicting with that expectation.

This isn't just a naming problem, but a communication problem. If I knew that Astra was the best at coding but a bit bad at writing, and Sol was almost as smart but better at technical writing, then that's the kind of thing I expect to be documented.

One possible way: A simple matrix with horizontal columns like "writing", "coding", "math", rows are the models, and each box is a number of stars from 1-5.

Instead, this sort of thing becomes lore, shared by word of mouth.

▲

tedsanders 3 hours ago | parent | next [-]

> Do you then recommend Sol for technical documentation?

I don't know, to be honest. Technical documentation is such a multidimensional skill, and I also don't know your budget. I'd imagine Terra might have the most readable prose, but Astra might be more likely to catch bugs or mistakes to highlight. If the documentation is about a more esoteric language or situation, Astra might have a greater edge than if it's a common language or situation. If you have a long list of documentation guidelines, that might also give more of an edge to Astra. I wouldn't want to make a recommendation without doing a lot of testing, and I haven't done a lot of testing.

▲

flockonus 25 minutes ago | parent [-]

Some food for thought, if someone from oAI can't make a direct recommendation for something as direct as "Technical documentation is such a multidimensional skill" - how could we know assess what model to use?

	▲	tedsanders 3 minutes ago \| parent \| next [-]
		To elaborate, I can certainly ask each model to write a doc about something I know well and tell you which output I like better. But to assess whether my preference on my problem is going to generalize across other people, other programming languages, other documentation types/formats, etc., and then assess statistical significance is a much broader question that I personally don't have the data to answer. I'd rather make no recommendation than a poorly informed recommendation.
	▲	scarab92 a minute ago \| parent \| prev [-]
		I use Astra 6 Extra High for everything. It might be overkill most of the time, but frankly all these models are “cheap”, and I’d rather just use a lot of compute than trying to discern which I’d best for ad hoc queries.

▲

lazharichir 3 hours ago | parent | prev | next [-]

> Put yourself in the user's shoes. On what basis should one pick a model for a given task?

Create a realistic eval suite for your use case using your actual data that you manually label, with easier, more complex, rarer samples, and run each model you are considering against it and decide based on the usual confusion matrix, cost, latency, or whatever metrics matter to you. There's really no other way to know – and you might be surprised that cheaper models can often perform your task(s) relatively well at a fraction of the cost.

	▲	awb 36 minutes ago \| parent [-]
		> Create a realistic eval suite for your use case using your actual data that you manually label, with easier, more complex, rarer samples, and run each model you are considering against it and decide based on the usual confusion matrix, cost, latency, or whatever metrics matter to you. I’d imagine there’s only a tiny percentage of users that can or want to do this.

▲

CSMastermind 3 hours ago | parent | prev | next [-]

> Do you then recommend Sol for technical documentation?

Every model I've tried is just abysmal at this, for what it's worth.

If any do it well please let me know and I'll give it a spin.

	▲	zenoprax 2 hours ago \| parent [-]
		My thoughts exactly. I recently experimented with some image generation models and I was surprised at how much context/prompting is needed to get something that doesn't look Lovecraftian. There was seemingly no downside to adding more and more detail up to the limit. Prose sits between code and images in terms of fuzziness and I wonder if promptmaxxing is the winning strategy for technical documentation. I've been thinking that fine-tuning a model to write differently shouldn't be that hard (tedious, perhaps) and if I had the resources I would consider doing it myself.

▲

ilaksh 4 hours ago | parent | prev | next [-]

You can start with Artificial Analysis benchmarks but, then you need to test in the real tasks anyway.

▲

saagarjha 3 hours ago | parent | prev [-]

I assume having a person write words for other people to read is not feasible?

▲

atombender 2 hours ago | parent [-]

Humans cannot be in the loop for autonomous work. I have agents review code and design/architecture documents, write alert messages based on rules, write analyses (e.g. outage incidents), and many other things. "Having a person write words" would mean not using agents, defeating the whole point.

I do not use agents to write on behalf of myself. I write my won technical write-ups and my own slide presentations. A lot of people don't; and it shows. There's much that agents can do better than humans now, but not everything.

▲

saagarjha 2 hours ago | parent [-]

This isn't technical documentation though, it's just notes the agents are keeping for each other. I think that is a very different task

	▲	atombender an hour ago \| parent [-]
		It's human-facing.