Remix.run Logo
▲ qalmakka 2 hours ago

A gut feeling I've been having since the start of the LLM era is that horizontal knowledge (aka, knowing that stuff exists, can be done in certain way, being able to understand the details without knowing them intimately and being able to do it with a wide array of things) will start getting more and more important the further we go in, because if you know that stuff exist you can point an LLM at it or get it to go in a specific way that it wouldn't take otherwise. LLM have a very strong tendency to just go from point A to point B, and they're often limited to what you can prompt.

This is basically the opposite of the last 50+ years where vertical knowledge (being a savant in a certain very specific field) was seen as the most important thing to have in a super specialised world.

This of course is just my perception, which needless to say, may be wildly off

▲kapep 2 hours ago | parent | next [-]

I'm so sure about that. LLMs seem to have that tendency when people ask them to solve a certain issue, but they are just as good for providing an overview of a topic - or even better, because details don't matter as much.

▲alex_smart 2 hours ago | parent | prev [-]

Isn’t the LLM already better at broadly knowing everything that exists than any human could ever hope to be?

▲PaulRobinson an hour ago | parent | next [-]

I’d say a SOTA LLM today has median depth of expertise over many, many disciplines, and slightly better than median expertise in a few key areas. “Jack of all trades, master of none”, springs to mind, but I’d say in the last 6 months it’s got above the median in some key areas.

The sweet spot is a true expert in a domain driving the LLM and keeping it honest, whether that be via agents/skills or reviewing work. In software dev, there’s huge tranches of work at junior/mid level where LLMs can just go, and anything where there is a clearly testable goal it can iterate on faster than a human expert, for example recent progresses in maths proofs.

But, I don’t think there’s an LLM out there that is an expert at everything. At least, not yet.

▲qalmakka 2 hours ago | parent | prev | next [-]

yep, but it's not better at picking strong, long term decisions in my experience. The LLM most of the time is like a very knowledgeable employee that just does what it's told, and if the boss doesn't understand that you can do something, will often not ask for it to begin with.

for instance, if you use a shitty library X in a project, and you ask your frontier model "please change A -> B" the model will just do that, it will not tell you "well maybe you could also swap the whole thing with Y" because that's not its job - it's job it to change A to B.

▲skeptic_ai an hour ago | parent | next [-]

That’s why you just ask: do you think this is a good idea? Correct me if I’m wrong

▲wolttam an hour ago | parent | prev [-]

So all it takes is asking broad “hey LLM, is there any obviously better approach to what we’re doing?” Questions once in a while.

(I agree, they may not be good at making strong long-term decision still, but that may also be an issue of inadequate context a lot of the time)

▲roveo an hour ago | parent | prev [-]

LLMs currently are a great demonstration of the wide difference between knowing things and applying them in practice. Knowledge contained in LLMs is vast, but latent.

Humans have a practically infinite "context window" (active memory), we make associations and vaguely remember that thing we heard about that might be useful here, let's explore this thread ("you're absolutely right!"). Agents, being powerful as they are, don't do that, they pattern-match on the task and use the obvious solution, but don't activate their latent knowledge until you point them in the right direction.

The obvious solution might be the best one in 80% of cases or if you don't really care to get the best results (and it's absolutely fine!), but if you do care in your particular case, you have to babysit your agents and really understand if they're doing what you had in mind.