> we can surely have the verifiers explain those

This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.

I think this is an artefact of how we train them, and they could be trained to be better at this if we focused on it, but at the same time, it shows its possible for these things to be able to do things they cant explain to us (granted, they can't explain them well - you can bash your way to understanding with them anyway but they aren't good at it)

▲

A_D_E_P_T 3 hours ago | parent | next [-]

> This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.

It's amazing to me that language models are still such poor writers and communicators. The mathematics community wrestled with this recently, when many solutions dropped into their laps but were not explained very well...

Thing is, (1) generic LLMs will improve, and (2) their lack of ability in this area has a lot to do with the rhetorical and structural habits they pick up during RLHF/instruction-tuning, which trained them for general-purpose comms. Zero-training base models can presumably be trained specifically to communicate scientific concepts. Interesting times.

(Edited to add: With respect to (1), compare Opus 5 to 5.5. 5 was a terrible communicator, even of software/coding concepts, whereas 5.5 is excellent and leaves very little to be desired.)

▲

GuB-42 an hour ago | parent | next [-]

LLMs are excellent writers in the sense that they get the spelling, grammar, syntax, etc... right. They are certainly way better than I am, and they can compete with the best writers considering how AI-generated prose has winned literary prizes.

Many people don't like the style, probably by association with slop and fatigue more than the style being fundamentally wrong.

Being a good communicator is an entirely different thing, you can have terrible spelling and grammar and still be good at getting ideas across. It is a common situation when the communicator in question is not using his native language.

That LLMs are so good at writing and so bad at communicating is not surprizing. The key to good communication is getting into the other people's mind, LLMs can't do that, because they "think" in a way that is completely alien to us. We use words to express our thoughts, thoughts that are based on a physical world model: touch, movement, sound, light, etc... LLMs have no idea about how the physical world work, everything the see is just words, they have no intuitive idea of, say, objects falling down, just that "falling" and "down" go well together. And for explaining things, they will go "someone asks for an explanation, explanations usually use these words, I will be using the same kind of words".

▲

teiferer 2 hours ago | parent | prev | next [-]

> 5.5 is excellent and leaves very little to be desired.

Hm, that may have to do with (faking) confidence. I've often enough with 5.5 still run into cases where it confidently explained something, at the surface it kinda made sense, but the more I dug and verified, the more it became apparent that it was entirely wrong with the most crucial core concepts. It has gotten to a point where you really need a domain expert though to realize this, which makes it tough for using it in areas where you are not an expert and don't have easy access to one either.

▲

Loquebantur 2 hours ago | parent [-]

You point at the exact problem here: people are so used to arguing from ignorance, they don't even notice.

Verification isn't knowledge. Far from it. You can verify statements without knowing why they're true. Variants of that concept are e.g. used in cryptology.

You need humans to truly understand (to know the context) in order to make correct judgements. Without actually knowing the context, you have no way to compare.

▲

A_D_E_P_T an hour ago | parent [-]

An infallible oracle tells Archimedes that the speed of light in vacuum is ~1.08 billion kilometers per hour. Archimedes, with the iron age tools and methods at his disposal, might be able to verify this and make some use of it himself. Maybe. In any case, what he has gained is knowledge itself.

The same goes for our own time. A powerful AI can tell us things at the edge of what we can grasp -- or beyond what we can grasp -- and it would nevertheless be knowledge that we possess. We can work to verify it, we can have it explained to us as though we were schoolchildren, or we can simply accept it as a static fact of the natural world.

In truth, we're going to have to learn to coexist with minds that are more powerful than ours. At least in some respects, if not in all respects. The good scenario is that they share information with us, and we receive it however we can. "Head in the sand" and "mutual disdain" are bad outcomes.

▲

Loquebantur an hour ago | parent [-]

No, you're wrong.

You might want to look up the actual definition of knowledge. It's a pretty interesting topic in epistemology really, far from your simplistic take.

You already live in coexistence with minds more powerful than yours. Society works in a way that you never need to notice that, those minds taking care of things beyond your grasp for you.

The downside of that "convenience" is you developing an over-simplified bubble-view of the world, rendering you prone to arbitrarily hazardous misjudgements.

Here, you align foolishly with interests that cannot possibly be your own. Namely, exploiting scientific knowledge for personalized monetary gain on an unsustainable scale.

When a tiny group of people hogs essentially all scientific knowledge, that renders you what?

Your idea of being graciously provided with some "basic income" or whatever is blind copium. You would live as livestock at best, to be discarded as useless whenever your masters see fit.

	▲	A_D_E_P_T 30 minutes ago \| parent [-]
		The dictionary definition says that knowledge is information gained by study -- i.e., in the very vast majority of cases, by being told. What's your definition? Come on, humor me. The rest of your post is excessively emotional and off-target.

▲

AlienRobot 3 hours ago | parent | prev [-]

To be fair, most people aren't writers, so the "average" writing the statistical model can produce is mediocre writing.

▲

RugnirViking 3 hours ago | parent | next [-]

I don't think this is the issue. Most people are mediocre mathematicians, too, yet the companies have selected well enough for math literacy in training data (presumably). I think its a matter of training - they are often post trained by other agents (which read everything), human reviewers not paid enough to care (easily wowed by 10 paragraph length essay with correct punctuation), and deterministic scripts (checking for presence of a specific point, so agents are incentivised to write as much as possible to have more chances of hitting the unknown thing the script is looking for)

As far as I know labs try to train the other way, else we'd have monstrously long and useless outputs, but its hard to overcome these natural biases for length

▲

teiferer 2 hours ago | parent | prev [-]

I don't think that's a suitable mental model of how LLMs work. Their output is not just the statistical average of all the training data. If it was then the output would be quite useless.

	▲	Loquebantur an hour ago \| parent [-]
		An LLM trained on the entire internet incorporates essentially all (human-known) aspects of things. Like a superposition of all kinds of writing styles, all perspectives on every topic and all kinds of ways to do things right and wrong. In post-training (and of course already by sensibly curating the input data in the first place), people try to select useful submanifolds from that. Ideally, you get a combination of all the extant strengths of people combined into one model. Self-improvement then leaves the sphere of human capability for good.

▲

jrumbut 4 hours ago | parent | prev [-]

And in many cases it is not really the new solution that is important. It is the new techniques, abstractions, or other sense-making or investigative tools that made the solution possible that matter.

And the abstractions do still matter, since in many cases it is an issue of tractability. Perhaps an LLM can force its way to a solution through 3 million mind numbingly boring and convoluted steps, but if the number of steps increases exponentially without the right abstraction that means it can only handle a slightly larger problem than a human with pen and paper.