A lot of effort is spent to make the "conversation" feel just like a human-to-human interaction. This is not a naturally occurring phenomenon due to the technology, but rather a feature carefully engineered by those companies in order to get people hooked. Then they have all these tiny nudges like the typing animations or the "thinking..." popups before the next chat message appears.

At some point you might have also noticed the over-use of emojis, the bolted-on jokes, and the tendency to always approve what the user says (even though they have toned that down after backslash). At some point too many people thought they were in a relationship with the chatbot, because it always encouraged and approved them, so they had to hotfix it.

It's a bunch of really dark psychological patterns that are carefully combined by very clever people in order to create the false illusion that the user is experiencing something deeper than an engineered simulation of human interaction.

I think the technology is really useful, but they are obviously not happy with simply replacing a google-like query interface, they want users to fall in love with the product and mentally treat it like a fellow human being - and that's what I think is insincere.

▲

ianm218 3 days ago | parent | next [-]

To get more concrete are you using coding agents like Claude Code/ Codex/ opencode etc? What kind of work are you doing specifically?

If you are doing the kind of median enterprise tech work these tools are just good enough to do it at a relatively high level or atleast heavily augment people doing it.

Examples would be like adding routine CRUD features to APIs/ improving observability/ adding tests or accessibility features to codebases etc.

	▲	bflesch 3 days ago \| parent [-]
		It try to explain it better in my longer sibling comment. I'm not using any coding agents. Their engineers can't be bothered to design their own webapp properly so I don't trust their binaries. For me both Claude and ChatGPT are query-response services and replacements for google. They help with error messages, single-file MVPs, and software design problems such as comparison of different modules. In my experience everything that goes beyond 200 lines creates issues down the line, so I try to keep interactions really short. Of course they can convincingly add CRUD functionality or tests, but one needs to double check their correctness, and if the subtle bugs are finally spotted then one needs to fix them anyways. It's good for a first draft but I wouldn't use agents on a codebase I actually care about. Unfortunately the billion-dollar funding forces the AI startups to make a return, and they are finding it in a vulnerable cohort of people who respond positively to a simulated human interaction, which is why they are focusing so much on it. The query-response knowledge interface was the moat of google, and nowadays it can be 80% replaced with a local GPU and an open model. They know it, which is why they try to hook people on the simulated human interaction aspect of their interfaces through chatbots and voice chat.

▲

pixl97 3 days ago | parent | prev | next [-]

>A lot of effort is spent to make the "conversation" feel just like a human-to-human interaction.

We'll in humans we call this an education and it takes quite a long time to get one.

▲

bflesch 3 days ago | parent [-]

Not a good comparison. Education is the part where they train on all digital content they can get their hands at, no matter the copyrights.

You get your education, you can replace google as a query-response interface to all digital content.

But then they use system prompts to simulate a fake persona and a user interface such as female voices or chat conversation in order to suggest that one is interacting with a real human being. This is clearly aimed at exploiting vulnerable cohorts of people, because the knowledge base part of this innovative technology is already solved.

Like casinos and social media companies, they know the profit is in the "whales" who can be psychologically manipulated to spend their time and money against their own interests.

	▲	pixl97 2 days ago \| parent [-]
		This is oddly conspiracy theory oriented. How would you program a LLM so it gives useful information to people with the least amount of people bitching about it? At the end of the day the LLM does not have a native persona. It has countless numbers of them. It can act like an autistic man, a flirty woman, a kid from some country you've never heard of. Bringing forth an agreeable persona from the myriad is a bad thing?

▲

fwipsy 3 days ago | parent | prev [-]

POV OpenAI, early 2021. You have a pretty good next-token predictor called GPT-3. You noticed sometimes it can do useful things if you write out the start of a task and let it predict the next step. However, sometimes it's very difficult to frame a task that way. So instead you train it to predict the answer based on a question or instructions. Oh wait, it didn't get it right the first time... better let users iterate too. Now you have a conversation.

Things like loading indicators are basic good UI dating back to the 90s.

A/B testing and generally following user preferences might still push towards the dynamic you're describing, as it did with gpt-4o. xAI and a few other companies like Replika also intentionally created "companion"/porn AIs. But in general, natural language was previously exclusive to humans. It's completely natural that the first technology capable of it would therefore be perceived as more human. It's worth trying to resist this tendency, but it doesn't require evil intent on the part of the creators.

▲

bflesch 3 days ago | parent [-]

The "conversation" interface is exactly the same workflow as one used with google search: user states a query, page loads, user adapts query because they are not happy with the result, until they end up with a suitable result which makes them close the tab.

So they have made this amazing query-response system which is far superior to google due to the summarization of query results from the global web and the auto-translation to present them in the user's native language. This is the type of raw query-response capability which many software engineers are trying to use in their agentic coding sessions.

However, after achieving such innovation, the AI startups consciously choose to apply social media KPIs to their query-response startup, which incentivizes all the dark patterns we have seen in their user interface. They notice that a certain subset of users can be tricked into believing that the startup's query-response interface has human-like qualities such as a name and persona.

This user cohort shows amazing metrics in terms of time spent on app, so they adapt their user interface and their system prompts accordingly. The AI startup doesn't have to care if the reason for humans accepting the illusion of a simulated human interaction is due to social circumstances (lack of emotional intimacy) or an underlying psychological vulnerability that the startup is actively exploiting.

The AI startup only cares if their "simulated human interaction" product receives negative attention from normal people who are not part of the vulnerable cohort, e.g. the suicides or the parasocial romantic relationships with the chatbots.

It is exactly the same as in the gambling industry: There is a certain subset of users called "whales" who are the cash cows for casinos, but if you look at the actual humans who are labeled with this term one can see pathological gamblers, most of which are ruining their lives and families. Casinos do everything to prevent people from jumping from their roofs after they lost all their money.

If AI startups can use simulated human interactions to make vulnerable people act against their own interests in the same way as casinos and social media companies do, it will allow them to make shitloads of money.

But if you're actually a clever person then be honest to yourself and others about what you are working on, and why these human-like features are really added to the user interfaces of OpenAI or Anthropic or the other AI startups.

So this is my framing of the situation.

I don't think this kind of problem can be overlooked by the insiders, and we might see some internal rifts along these lines: Will our AI startup simulate a human interaction in order to exploit our vulnerable peers, or will our AI startup focus on delivering the best response to our user's queries?

Because now we have local models, which - assuming one has suitable hardware - provide 80% of the utility in terms of a query-response knowledge base.

As we are currently seeing, the AI startups with billion-dollar funding have very big economic incentives to focus on the "simulated human interaction" part of the equation, because their investors need returns.

The biggest strategic blunder I see at Google. Because if Google actually changes their excellent query-response user interface to a chat conversation which simulates a human interaction with persona, name, and voice, then they knowingly pivot to the same social media KPI driven business as OpenAI and Anthropic are struggling with.

	▲	fwipsy 2 days ago \| parent [-]
		Feels like OpenAI and Anthropic are both more interested in B2B. OpenAI I'm more suspicious of after GPT-4o. I mostly use Claude and I haven't noticed anything that feels like an intentional dark pattern. It's constantly reminding me that it's a chatbot.