I am seeing a lot of people suffering from AI psychosis getting interested in writing a trading bot using large language models. They give them access to basic market data and then ask the model for analysis of the current market conditions. The bot talks about seeing high volume here, that being a confirmation for this, and that doing an invalidation of that etc. I looked into it after someone pitched me theirs, only to realize it doesn't even have access to the trading volume. So it was hallucinating a whole narrative about that and the guy shilling this to me hadn't noticed it didn't have access to trade volume yet coming up with a whole rationale for what it is observing.

It boggles the mind what AI is doing to some people’s brains. A few months ago we released various vibe-coded SDKs in few languages and a CLI for a rest API we had. Then an immediate ask after that was a dashboard that shows the usage of those various SDKs and CLI. I thought it would be straightforward to check the userAgent on all the API calls, but realized that all the user agents across all SDKs and CLI were the same. So I made a couple of simple PRs to all of them to have the language name in the user agents, and was composing a reply about how we only have data for the overall new SDKs/CLI usage but once these PRs are merged and released we will have the breakdown. Before I could send my reply, I got a reply from another fairly senior dude saying “Here is a dashboard for the usage” with fairly convincing and organic-looking usage pattern for all the SDKs and all sorts of breakdowns and charts etc.

I was confused, how did he figure it out? Is it somewhere I missed? I checked the query in the dashboard and it was this insane looking query that was correctly looking for the shared userAgent, but also doing a shit ton of odd functions and weird looking math. I asked the guy and he pointed me to the obligatory 1,400 line markdown file describing the “plan” and “approach”. In between all the raw LLM-isms and very sophisticated “weighted normalized distribution of language spread over time” sounding sentences, there was a sentence that said “Since we only have one userAgent for all tools, we use industry standard distribution of language popularity to approximate the usage, and apply some randomness to emulate an organic usage pattern”

So basically the AI just made up some numbers that look plausible (python 20%, JavaScript 30%, CLI 25%, etc) with some randomness thrown in “to look organic”, and the guy was like “that checks out”. When I pointed that these numbers are wrong because X, Y and Z, the person who asked for the data said “that’s ok, it’s a great start and gives me something to use for planning with other teams. Feel free to refine them if you want, but it’s good we have some numbers”. I’m planning my exit soon.

I found a scam online about 18 months ago that was selling access to some "AI trading platform" that was supposed to have the kind of bot you mention. I used the live chat to talk to someone and asked about demonstrable proof - like given data up through the end of last month, make picks for this month, then look at this month's data to see how it would have done.

"Then this product is not for you." Right, not for anyone thinking critically about how any of this works.