Maybe. With enough money and resources and time, maybe. But it would need to be very reliable to be at all useful, and that level of reliability is very hard to achieve with something so nuanced and complex.

You have an entire lifetime of experience and the wisdom of countless generations to inform you of the context with which you should process new information. You were specifically trained for years to be able to do this somewhat effectively. Doing the same for an LLM may or may not be possible. They contain much of that experience in some form, but are not well equipped to make use of it iteratively. It is incredibly expensive to attempt even a simple and ineffective version of it with our current methods.

If the LLM can achieve even 90% accuracy, it becomes more of a hindrance than a help. Do you make use of the function and trust it successfully understood the provenance of new information? If so, ten percent of the time you will be misled, which could be disastrous across a population. If you don't simply trust it, then do you painstakingly check the provenance of everything it tells you? Of course not, you already don't do that with current search engines and blogs and such. Yet falling somewhere in between would be so much worse than simply not having the LLM attempt to discern trustworthiness from providence in the first place.

But all of that is opinion and I may be incorrect. Perhaps someone will create some ingenious way of imparting wisdom and discernment to LLMs next year and this will all be trivial.

Nothing stopping us except perverse incentives. Why would a company admit its model isn't using 100% reliable data?

It has little to do with gullibility and more to do with the way they do things.

In principle AI Agents are superhuman at codeforces but tons of these vibe coded apps have ridiculously stupid performance problems.

The problem is a utilization of expertise thing, it can tell you with incredible detail about pros and cons and caveats to a problem if you ask about it. Truly superhumanly intelligent.

Calibration when it comes to actual decision making takes an incredible reduction in capability after that. Maybe it has something to do with the sycophancy or something but its like someone with exactly zero opinions (and therefore, taste).

My experience is that you can brainstorm with it on "Do A or B?" and it will produce a coinflip answer A, then if you continue and ask about a specific concern "what about this aspect of B?", it immediately 180s and flips back to B being way better than A.

Acting on this expertise when decision making points arise naturally for itself is where everything really nosedives. It's just gonna do whatever.

So when you are doing this "financial advice please" questioning, it is entirely hitting these weaknesses squarely in the face.

Of course, and I'm really not trying to move the goal posts here, but I agree that that's what (most) humans are going to do. But we're rapidly moving in a direction of agent-to-agent flow and "they" seem to lack the degree of discernment the average human does. They certainly don't have the same incentive structure we do to detect fraud. They happily regurgitate and build off of lies because, to them, the lies are not lies, and they can't be. They're just a pile of geometrically close labels from the perspective of the model.

So “caveat emptor”? I get that’s always part of reality but we shouldn’t exactly steer into that skid IMO. It just shifts the burden to the rest of us and does nothing to address the root cause

Google search results are amazing and you know it. It’s 100.

(I am not including AI results in this).