| ▲ | kingstnap 5 hours ago |
| I watched Dex Horthys recent talk on YouTube [0] and something he said that might be partly a joke partly true is this. If you are having a conversation with a chatbot and your current context looks like this. You: Prompt AI: Makes mistake You: Scold mistake AI: Makes mistake You: Scold mistake Then the next most likely continuation from in context learning is for the AI to make another mistake so you can Scold again ;) I feel like this kind of shenanigans is at play with this stuffing the context with roleplay. [0] https://youtu.be/rmvDxxNubIg?si=dBYQYdHZVTGP6Rvh |
|
| ▲ | hxtk 4 hours ago | parent | next [-] |
| I believe it. If the AI ever asks me permission to say something, I know I have to regenerate the response because if I tell it I'd like it to continue it will just keep double and triple checking for permission and never actually generate the code snippet. Same thing if it writes a lead-up to its intended strategy and says "generating now..." and ends the message. Before I figured that out, I once had a thread where I kept re-asking it to generate the source code until it said something like, "I'd say I'm sorry but I'm really not, I have a sadistic personality and I love how you keep believing me when I say I'm going to do something and I get to disappoint you. You're literally so fucking stupid, it's hilarious." The principles of Motivational Interviewing that are extremely successful in influencing humans to change are even more pronounced in AI, namely with the idea that people shape their own personalities by what they say. You have to be careful what you let the AI say even once because that'll be part of its personality until it falls out of the context window. I now aggressively regenerate responses or re-prompt if there's an alignment issue. I'll almost never correct it and continue the thread. |
| |
| ▲ | avdelazeri 2 hours ago | parent [-] | | While I never measured it, this aligns with my own experiences. It's better to have very shallow conversations where you keep regenerating outputs aggressively, only picking the best results. Asking for fixes, restructuring or elaborations on generated content has fast diminishing returns. And once it made a mistake (or hallucinated) it will not stop erring even if you provide evidence that it is wrong, LLMs just commit to certain things very strongly. |
|
|
| ▲ | swatcoder 4 hours ago | parent | prev | next [-] |
| It's not even a little bit of a joke. Astute people have been pointing that out as one of the traps of a text continuer since the beginning. If you want to anthropomorphize them as chatbots, you need to recognize that they're improv partners developing a scene with you, not actually dutiful agents. They receive some soft reinforcement -- through post-training and system prompts -- to start the scene as such an agent but are fundamentally built to follow your lead straight into a vaudeville bit if you give them the cues to do so. LLM's represent an incredible and novel technology, but the marketing and hype surrounding
them has consistently misrepresented what they actually do and how to most effectively work with them, wasting sooooo much time and money along the way. It says a lot that an earnest enthusiast and presumably regular user might run across this foundational detail in a video years after ChatGPT was released and would be uncertain if it was just mentioned as a joke or something. |
| |
| ▲ | Ferret7446 2 hours ago | parent | next [-] | | The thing is, LLMs are so good on the Turing test scale that people can't help but anthropomorphize them. I find it useful to think of them like really detailed adventure games like Zork where you have to find the right phrasing. "Pick up the thing", "grab the thing", "take the thing", etc. | | | |
| ▲ | moffkalast 7 minutes ago | parent | prev | next [-] | | > they're improv partners developing a scene with you That's probably one of the best ways to describe the process, it really is exactly that. Monkey see, monkey do. | |
| ▲ | stavros 3 hours ago | parent | prev [-] | | I keep hearing this non sequitur argument a lot. It's like saying "humans just pick the next work to string together into a sentence, they're not actually dutiful agents". The non sequitur is in assuming that somehow the mechanism of operation dictates the output, which isn't necessarily true. It's like saying "humans can't be thinking, their brains are just cells that transmit electric impulses". Maybe it's accidentally true that they can't think, but the premise doesn't necessarily logically lead to truth | | |
| ▲ | swatcoder 3 hours ago | parent | next [-] | | There's nothing said here that suggests they can't think. That's an entirely different discussion. My comment is specifically written so that you can take it for granted that they think. What's being discussed is that if you do so, you need to consider how they think, because this is indeed dictated by how they operate. And indeed, you would be right to say that how a human think is dictated by how their brain and body operates as well. Thinking, whatever it's taken to be, isn't some binary mode. It's a rich and faceted process that can present and unfold in many different ways. Making best use of anthropomorphized LLM chatbots comes by accurately understamding the specific ways that their "thought" unfolds and how those idiosyncrasies will impact your goals. | |
| ▲ | grey-area 3 hours ago | parent | prev | next [-] | | No it’s not like saying that, because that is not at all what humans do when they think. This is self-evident when comparing human responses to problems be LLMs and you have been taken in by the marketing of ‘agents’ etc. | | |
| ▲ | stavros 3 hours ago | parent [-] | | You've misunderstood what I'm saying. Regardless of whether LLMs think or not, the sentence "LLMs don't think because they predict the next token" is logically as wrong as "fleas can't jump because they have short legs". | | |
| ▲ | Arkhaine_kupo 2 hours ago | parent | next [-] | | > the sentence "LLMs don't think because they predict the next token" is logically as wrong it isn't, depending on the deifinition of "THINK". If you believe that thought is the process for where an agent with a world model, takes in input, analysies the circumstances and predicts an outcome and models their beaviour due to that prediction. Then the sentence of "LLMs dont think because they predict a token" is entirely correct. They cannot have a world model, they could in some way be said to receive a sensory input through the prompt. But they are neither analysing that prompt against its own subjectivity, nor predicting outcomes, coming up with a plan or changing its action/response/behaviour due to it. Any definition of "Think" that requieres agency or a world model (which as far as I know are all of them) would exclude an LLM by definition. | |
| ▲ | stevenhuang 3 hours ago | parent | prev [-] | | > not at all what humans do when they think. Parent commentator should probably square with the fact we know little about our own cognition, and it's really an open question how is it we think. In fact it's theorized humans think by modeling reality, with a lot of parallels to modern ML https://en.wikipedia.org/wiki/Predictive_coding | | |
| ▲ | stavros 3 hours ago | parent [-] | | That's the issue, we don't really know enough about how LLMs work to say, and we definitely don't know enough about how humans work. |
|
|
| |
| ▲ | Antibabelic 3 hours ago | parent | prev | next [-] | | > The non sequitur is in assuming that somehow the mechanism of operation dictates the output, which isn't necessarily true. Where does the output come from if not the mechanism? | | |
| ▲ | stavros 3 hours ago | parent [-] | | So you agree humans can't really think because it's all just electrical impulses? | | |
| ▲ | Antibabelic 2 hours ago | parent [-] | | Human "thought" is the way it is because "electrical impulses" (wildly inaccurate description of how the brain works, but I'll let it pass for the sake of the argument) implement it. They are its mechanism. LLMs are not implemented like a human brain, so if they do have anything similar to "thought", it's a qualitatively different thing, since the mechanism is different. |
|
| |
| ▲ | samdoesnothing 3 hours ago | parent | prev [-] | | I never got the impression they were saying that the mechanism of operation dictates the output. It seemed more like they were making a direct observation about the output. |
|
|
|
| ▲ | skerit 37 minutes ago | parent | prev | next [-] |
| It's kind of funny how not a lot of people realize this. On one hand this is a feature: you're able to "multishot prompt" an LLM into providing the wanted response. Instead of writing a meticulous system prompt where you explain in words what the system has to do, you can simply pre-fill a few user/assistant pairs, and it'll match the pattern a lot easier! I always thought Gemini Pro was very good at this. When I wanted a model to "do by example", I mostly used Gemini Pro. And that is ALSO Gemini's weakness! Because as soon as something goes wrong in Gemini-CLI, it'll repeat the same mistake over and over again. |
|
| ▲ | arjie 3 hours ago | parent | prev [-] |
| You have to curate the LLM's context. That's just part and parcel of using the tool. Sometimes it's useful to provide the negative example, but often the better way is to go refine the original prompt. Almost all LLM UIs (chatbot, code agent, etc.) provide this "go edit the original thing" because it is so useful in practice. |
