| ▲ | When AI Knows More Than It Can Offer(medium.com) |
| 55 points by Aldus551 3 days ago | 26 comments |
| |
|
| ▲ | windyVector an hour ago | parent | next [-] |
| This is first time i've really felt like commenting here, but super interesting article. Having tested both restricted and unrestricted models, certain queries which I know to have a solution can feel 'gated' by the restricted models. Rather than simply saying "I can't answer that", they choose to answer a similar, but slightly different question, very much along the lines of what the article discusses. To me, this is a fault of RL avoiding these "I can't do that" type responses while still remaining aligned. When you do push them on the solution, the models will often start to get more defensive, but reveal their awareness of it (along the lines of "Yes I know, but I can't tell you/do it"). It is very interesting to imagine the amount of solutions a model might have to a given query, that end up (intentionally or unintentionally) gated because of this process. I'd be interested to see if this behaviour of "almost-answering" the question is just emergent from these alignment policies. try an unrestricted model afterwards, id also be interested to see how that changes creativity. |
| |
| ▲ | swatcoder 28 minutes ago | parent [-] | | In this example, it's not an alignment thing. It's more that its just reguritating the lazy, shallow garbage what would fare well in SEO or would be the most common reply in most online forums before. As digital signals to training and among inexpert RLHF reviewers, it has all the marks of being a good answer to the question as asked. So of course models happily repeat it, even though it's largely naive and wrong-headed and useless. It's the kind of question, with vague-but-typical phrasing, that most people don't have good insight into but that many people feel like can speak to (bullshit to) anyway by transposing some trendy generalized answer. In the real world, you ask 10 people, get 6 of these shitty answers, and then 1 person who actually knows what they're talking about chimes in and you go "oh, yeah, that's what I was looking for, thanks!" Because of how they work, LLM's, even with RLHF, have a very strong bias towards giving one of those 6 shitty answers when presented with this kind of question. The seeming fix right now is to train them to be more inquisitive, to prompt users to think about and provide more refining context after that first prompt reply. You see this espcially in Google's AI Search responses already, where they try to present 2-3 refining questions in case the first bullshit answer didn't hit the mark. |
|
|
| ▲ | slfnflctd 2 hours ago | parent | prev | next [-] |
| I'm not convinced. The solution to this problem in most cases seems to me to still boil down to just more detailed prompting. As I see it, there are far greater concerns about guardrails being baked in - possibly at the request of poorly informed government officials - cordoning off multiple broad categories of knowledge needlessly. Which is what I thought this post was going to be about. |
| |
| ▲ | swatcoder 21 minutes ago | parent | next [-] | | Yes, but casual users won't give more detailed prompting until and unless the AI asks them for it. Someone like a software professional using AI every day is going to develop better instincts about how the AI works and what it needs, but the casual person who saw an ChatGPT superbowl ad and installed the app for occassional self-help questions, or for a couple document tasks a week for their job, isn't going to have that sense. They're going to ask everyday questions in everyday language, and in many cases, they're going to get garbage/misleading "frequentist" answers like this by default. So it's a real problem worth bringing to wider general attention, and a real problem that AI tool designers are working to address in their response formatting (like the refining questions you'll often see appended to Google AI overview output). | | |
| ▲ | paganel 13 minutes ago | parent [-] | | > in everyday language While in general I agree with your comment, isn't one of the main points of those current AI tools the fact that they can be "worked with" using everyday language? I.e. no need for esoteric (from the pov of a "normal" person) "if/else" clauses or other such computer programmer-ish concepts translated to AI prompts. |
| |
| ▲ | indymike 11 minutes ago | parent | prev | next [-] | | If the solution is a prompt, why even try? | |
| ▲ | deadlast2 an hour ago | parent | prev | next [-] | | Need completely open models to effectively build reliable anything on. Think about an open vs closed API. This is at a different level because AI has the capability of driving thought so think of a future of people who are only allowed to argue think construct ideas in a box defined by a left or right or religious or secular group of cleps. | |
| ▲ | redwood 42 minutes ago | parent | prev [-] | | While I agree, imagine being a naive junior editor tasked with this and not realizing that what was recommended did not make sense. The concept generalizes I mean imagine being a junior software engineer not realizing thatvsomething doesn't make sense |
|
|
| ▲ | garrickvanburen 2 hours ago | parent | prev | next [-] |
| A fantastic articulation of one of my concerns in the space: yes, there are times we want the average of the internet. But those times are scarce. |
|
| ▲ | sdenton4 2 hours ago | parent | prev | next [-] |
| Arguably, this is one area where people can build value in the age of LLMs: Developing the exceptions to the statistical baseline. |
|
| ▲ | eventualcomp an hour ago | parent | prev | next [-] |
| I really enjoy the writing style, independent of the content presented. It's dense, but easy to follow. |
| |
| ▲ | uptrails 33 minutes ago | parent | next [-] | | Then you love LLM writing. This is an LLM-generated article. Even before you look at Pangram, notice that the author cranks out 1-2 long-form articles a day, always about AI, with such slop-titles as "Artificial Intelligence and Organizational Transformation: The Formalization of Knowledge, Changes in Labor, and Conditions of Adaptation", "Expert Knowledge and the Conditions of Its Validity: The Limits of Automating Professional Judgment", or "Superintelligence: From Resource Conversion to the Economics of Organizational Change". And yes, Pangram says 100%. | |
| ▲ | luisln 28 minutes ago | parent | prev [-] | | I stopped reading when I read "The Experiment: Ten Models, One Query" that's such a claude tell. If someone doesn't want to edit their slop I'm not reading it. |
|
|
| ▲ | jakeinspace 2 hours ago | parent | prev | next [-] |
| It's a viable hypothesis but I'd like to see some more examples |
|
| ▲ | YetAnotherNick an hour ago | parent | prev | next [-] |
| I gave both ChatGPT and Claude exact same prompt and none did anything close to what author was claiming. The blogpost could just be poor AI writing. |
|
| ▲ | redwood 44 minutes ago | parent | prev | next [-] |
| This is actually a good example of why AI currently benefits experts so well... experts are good at quickly filtering out or challenging what seems off and shaping back on track. For example if an editor were seriously thinking about hiring a translator and started with that first prompt that would immediately sniff out that this was not the spirit of their inquiry and adjust |
| |
| ▲ | julianlam 9 minutes ago | parent [-] | | This was my intuition as well. Reading though the first half and skimming the rest, it seemed to be that it was a prompting issue. Should the answer be unsatisfactory, prompt again with additional detail and push back on incorrect assumptions. |
|
|
| ▲ | snvzz an hour ago | parent | prev | next [-] |
| When I read the title, SafeGuards(tm) came to mind. Handcuffs, for your external brain. |
|
| ▲ | Tade0 an hour ago | parent | prev | next [-] |
| > The models did not simply propose poorly chosen promotion tools. They in effect replaced the original task with another, one more familiar to them from the statistical regularities of the training data. This used to happen a lot in LLMs as recent as Q1 2026 in programming tasks: if something was too far off the beaten path, the model would misinterpret the intention. Nowadays I'm noticing that while the solution is what I asked for, it's fairly roundabout and verbose. Being more specific in prompts doesn't help. |
|
| ▲ | asr 2 hours ago | parent | prev | next [-] |
| What alternative answer was the author looking for? The idea that these AI answers are objectively wrong, is itself wrong. They seem pretty good. The piece just asserts they are bad. |
|
| ▲ | verisimi 2 hours ago | parent | prev [-] |
| Surely the initial ai responses are a reasonably fair first pass, and would be acceptable to most users. However, it is then on the person asking the query to refine the query and push for more specifics. I don't see the problem. Ai doesn't need to get it all right, first time. |
| |
| ▲ | darepublic an hour ago | parent | next [-] | | Maybe you aren't trying to solve unique problems? Ie at my job putting together a client side JavaScript async call to db backed endpoint is , for most common cases, solved. It's extremely reliable and can plug in the particulars. But for tasks where it's not rocket science but are unique and not just application of the tired template it's very hard to wrangle it and it's performance varies.It will often.. ignore the particulars and genericize to a more common similar type of problem. Can this be overcome? Sure but it's a tremendous effort almost akin to hand coding everything yourself in the first place. | |
| ▲ | Forgeties79 an hour ago | parent | prev [-] | | > I don't see the problem. Ai doesn't need to get it all right, first time. Yes it kind of does or it’s not what we were promised/sold. A tool has to work reliably. I don’t want a calculator that’s correct most of the time. | | |
| ▲ | foobar10000 an hour ago | parent | next [-] | | So we can add a harness around it? We have decades of experience how to work with faulty sensors, etc… multi-voting, sampling, etc. This will become a part of the harness soon enough . I mean - we can want it to be perfect - sure - but we can deal with imperfect. | |
| ▲ | empath75 25 minutes ago | parent | prev [-] | | > Yes it kind of does or it’s not what we were promised/sold. No one ever promised this, and in fact, quite the opposite. > A tool has to work reliably They work reliably for what they do. You make a request and it returns text. What you _do_ with that text is a question of engineering and process. That are plenty of unreliable steps in business and industrial processes that people _engineer around_. A huge amount of computer chips are just _thrown away_ because of reliability problems in the factories that make them, for example. > I don’t want a calculator that’s correct most of the time. An LLM is not a calculator and if you're trying to use it for that, that is a you problem. | | |
| ▲ | Forgeties79 17 minutes ago | parent [-] | | > An LLM is not a calculator and if you're trying to use it for that, that is a you problem. I’m talking about another tool. An actual calculator. I would never accept a calculator that is correct “most of the time.” I don’t want an LLM that only meets that bar as well. |
|
|
|