This is a reasonable analysis. It explains where AI is useful but I think it doesn't touch on AI's trustworthyness. When data is good, AI may or may not be trusted to complete it's task in an accurate manner. Often it can be trusted.

But sometimes good data is also bad data. HIPAA compliance audit guides are full of questions that are appropriate for a massive medical entity and fully impossible to answer for the much more common small medical practice.

No AI will be trained to know the latter is true. I can say that because every HIPAA audit guide assumes that working patient data is stored on practice-owned hardware - which it isn't. Third parties handle that for small practices.

For small med, HIPAA audit guides are 100 irrelevant questions that require fine details that don't exist.

I predict that AI won't be able overcome the absurdities baked into HIPAA compliance. It can't help where help is needed.

But past all that, there is one particularly painful issue with AI - deployment.

When AI isn't asked for, it is in the way. It is an obstacle to that needs to be removed. That might not be awful if MS, Google, etc didn't continually craft methods to make that as impossible as possible. It smacks of disdain for end users.

If this one last paragraph wasn't endlessly true, AI evangelists wouldn't have so many premade enemies to face - and there would be less friction all around.

No, AI is just garbage. I asked AI a clear cut question about battery optimization in zen. It told me it's based on chrome, but it's based on firefox.

Ask it about a torque spec for your car? Yup, wrong. Ask it to provide sources? Less wrong but still wrong. It told me my viscous fan has a different thread than it has. Would I have listened, I would've shredded my thread.

My car is old, well documented and widely distributed.

Doesn't matter if claude or chatgpt. Don't get me started on code. I care about things being correct and right.

A couple weeks ago I was working on a parser that needed to handle a new file format that was a large change from existing formats. I wanted some test inputs, both valid and invalid cases. I had the codebase of a tool chain that I knew could generate valid files, some public documentation about the new file format, and my parser codebase.

A good problem to throw at AI, I thought. I handed the tools to a SOTA model and asked it to generate me some files. Garbage. Some edits to the prompts and I still get garbage. Okay, that's pretty hard to generate a binary with complex internal structure directly. Let's ask it to tell me how to make the toolchain generate these for me. It gives me back all sorts of CLI examples. None work. I keep telling it what output I am getting and how it differs from what I want. Over and over it fails.

I finally reach out to somebody on the toolchain team and they tell me how to do it. Great, now I can generate some valid files. Let's try to generate some invalid ones to test error paths. I've got a file. I've got the spec. I ask the LLM to modify the file to break the spec in a single way each time and tell me which part of the spec it broke each time. Doesn't work. Okay. I ask it to write me a python program that does this. Works a little bit, but not consistently and I need to inspect each output carefully. Finally I throw my files into a coverage guided fuzzer corpus and over a short period of time it's generated inputs that have excellent branch coverage for me.

What would effective have looked like to you in this situation?

I'll pick a few concrete tasks: Building a substantially faster protobuf parser, building a differentiable database, and building a protobuf pre-compression library. So far, AI's abilities have been:

1. Piss-poor at the brainstorming and planning phase. For the compression thing I got one halfway decent idea, and it's one I already planned on using.

2. Even worse at generating a usable project structure or high-level API/skeleton. The code is unusable because it's not just subtly wrong; it doesn't match any cohesive mental model, meaning the first step is building that model and then figuring out how to ram-rod that solution into your model.

3. Really not great at generating APIs/skeletons matching your mental model. The context is too large, and performance drops.

4. Terrible at filling in the details for any particular method. It'll have subtle mistakes like handling carryover data at the end of a loop, but handling it always instead of just when it hasn't already been handled. Everything type checks, and if it doesn't then I can't rely on the AI to give a correct result instead of the easiest way to silence the compiler.

5. Very bad at incorporating invariants (lifetimes, allocation patterns, etc) into its code when I ask it to make even minor tweaks, even when explicitly promoted to consider such-and-such edge case.

6. Blatantly wrong when suggesting code improvements, usually breaking things, and in a way you can't easily paper over the issue to create something working "from" the AI code.

Etc. It just wasn't well suited to any of those tasks. On my end, the real work is deeply understanding the problem, deriving the only possible conclusions, banging that into code, and then doing a pass or three cleaning up the semicolon orgasm from the page. AI is sometimes helpful in that last phase, but I'm certain it's not useful for the rest yet.

My current view is that the difference in viewpoints stems from a combination of the tasks being completed (certain boilerplate automation crap I've definitely leaned into AI to handle, maybe that's all some devs work on?) and current skill progression (I've interviewed enough people to know that the work I'm describing as trivial doesn't come naturally to everyone yet, so it's tempting to say that it's you holding your compiler wrong rather than me holding the AI wrong).

Am I wrong? Should AI be able to help with those things? Is it more than a ~5% boost?

Personally, everyone I’ve seen using AI either clearly didn’t understand what they were doing (in a ‘that’s not doing what you think it’s doing’ perspective), often in a way that was producing good sounding garbage, or ended up rewriting almost all of it anyway to get the output they actually wanted.

At this point I literally spend 90% of my time fixing other teams AI ‘issues’ at a fortune 50.

Once again. Replies only proving me right. Desperately trying to justify “ai bad I’m superior” mentality.