I do have an entirely AI written project, and I don't think I've reviewed even 1% of the code the bots have generated; it's been my pure vibe project. It's about what you'd expect, too.

Elsewhere, I still haven't reviewed the vast majority of AI code I've generated, but all of the AI-generated code I've submitted I did review exhaustively. I have a couple of past roles to thank for drilling some strong code-review strategies and - thus far - have had the willpower to reject an AI generation when I know I can't/won't diligently review.

LLMs do not do anything but generate a token. It's a trivial but critical distinction like RNG vs pRNG. The LLM doesn't learn, it doesn't do backtracking, their output is deterministic.

Not "chatgpt.com"s output, but the LLM powering it; Anthropic aren't customizing a version of Claude's weights/tensors on your inputs, they're adding billable input tokens to the baseline of your subsequent contexts.

Its very financially convenient for Anthropic and OpenAI if people think of the LLM as doing the work, because for a lot of people that conjures a sense of a system that inherently learns and improves.

Even some of the people I've worked with on low-level LLM harnessing tools/systems since early 2023 fall for it and start thinking of the LLM as an AI with the connotations of back-propagation, weight adjustment, ... learning.

The "AI" is in what the harness software does with the inference output and the context it feeds back to it.

It took forever to convince people that quality degrades with context length but then all we got was compaction, there's still huge resistance to harnesses actively curating the context.

A lot of non-devs I've spoken to assumed when their tool said "Now I need to re-read" it was overwriting the 'in memory' copy.

You open aider or code or claude and ask it to fix a bug in file.<c|cpp|js|py>, it reads the entire 8k file, reasons about it (in which it sometimes echoes specific lines of code), and then it "edits the file".

To edit a file, inference currently injects a stream of tokens into the context that your software identifies as a tool call.

[tool name="edit" file="..."] @120,131 - halt_and_catch_fire() + exit_nicely() [/tool]

And frequently the tooling will guide the model to re-read the file, in some pathological cases by prompting the model to generate a tool instruction to do it.

Your context now contains: pre-prompting, your prompt, first file read (8k tokens), reasoning, all the tool calls to make the changes, re-read (8k tokens), conclusory reasoning, tooling prompts llm to describe what has been done, work-done-summary.

Because of the way the underlying LLM works and because of the way the APIs are presented, tools are discouraged from redacting or eliding the original read.

Otherwise, what they'd already be doing is curating the context: read file once, append output from inference/tool calls, and when it's done, replace the original insert of the file.

Having 2 or more disagreeing versions of a significant source file in your context is detrimental to model attention and output quality, and you either see people who get it and manage it, or who wallow in superstition and hand-waving and a near eagerness to pay for "mistakes" that the "model is making" because they believe it is learning, on the fly.