Of course. But when you see text in the descriptor of a satchel on a satchel seller's site, you contextualize it as marketing. If you saw the same text in a textbook on the history, design, and manufacturing of satchels, you would contextualize it very differently.

When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.

I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.

▲

cortesoft 2 hours ago | parent | next [-]

There is nothing stopping us from adding the ability to understand the context of information an LLM consumes, too, right? The same process humans use to assess data reliability could be taught to an LLM?

▲

cure_42 2 hours ago | parent | next [-]

Maybe. With enough money and resources and time, maybe. But it would need to be very reliable to be at all useful, and that level of reliability is very hard to achieve with something so nuanced and complex.

You have an entire lifetime of experience and the wisdom of countless generations to inform you of the context with which you should process new information. You were specifically trained for years to be able to do this somewhat effectively. Doing the same for an LLM may or may not be possible. They contain much of that experience in some form, but are not well equipped to make use of it iteratively. It is incredibly expensive to attempt even a simple and ineffective version of it with our current methods.

If the LLM can achieve even 90% accuracy, it becomes more of a hindrance than a help. Do you make use of the function and trust it successfully understood the provenance of new information? If so, ten percent of the time you will be misled, which could be disastrous across a population. If you don't simply trust it, then do you painstakingly check the provenance of everything it tells you? Of course not, you already don't do that with current search engines and blogs and such. Yet falling somewhere in between would be so much worse than simply not having the LLM attempt to discern trustworthiness from providence in the first place.

But all of that is opinion and I may be incorrect. Perhaps someone will create some ingenious way of imparting wisdom and discernment to LLMs next year and this will all be trivial.

▲

Zarathustra30 2 hours ago | parent | prev [-]

Nothing stopping us except perverse incentives. Why would a company admit its model isn't using 100% reliable data?

	▲	cortesoft an hour ago \| parent [-]
		If real people are using the model for real, they will discover for themselves that the data isn’t reliable, and then might switch to a different model. The incentive will be to maintain a subscriber base.

▲

jonasced an hour ago | parent | prev | next [-]

I agree and finding "official domains" is an interesting problem in and of itself. I wonder if PageRank like weights of input data would help, or some web of trust-like idea to prune blatant spam websites.

▲

tsunamifury 4 hours ago | parent | prev | next [-]

Yea if you ever want to test how gullible even latest SOTA models are ask it for financial advice on a specific IPO and watch it vaccume up sell side nonsense and spew it out back to you as absolute fact.

	▲	kingstnap an hour ago \| parent [-]
		It has little to do with gullibility and more to do with the way they do things. In principle AI Agents are superhuman at codeforces but tons of these vibe coded apps have ridiculously stupid performance problems. The problem is a utilization of expertise thing, it can tell you with incredible detail about pros and cons and caveats to a problem if you ask about it. Truly superhumanly intelligent. Calibration when it comes to actual decision making takes an incredible reduction in capability after that. Maybe it has something to do with the sycophancy or something but its like someone with exactly zero opinions (and therefore, taste). My experience is that you can brainstorm with it on "Do A or B?" and it will produce a coinflip answer A, then if you continue and ask about a specific concern "what about this aspect of B?", it immediately 180s and flips back to B being way better than A. Acting on this expertise when decision making points arise naturally for itself is where everything really nosedives. It's just gonna do whatever. So when you are doing this "financial advice please" questioning, it is entirely hitting these weaknesses squarely in the face.

▲

cole_h06 4 hours ago | parent | prev [-]

[flagged]