> A lot of people here seem to be living in a different universe than me

I feel this way on this topic too.

> I think detractors believe you should just let AI do the job blindly instead of leveraging it as a tool to accelerate you.

The problem is; how _should_ I use AI? On a previous thread, I had two replies to the same comment, one saying "provide the LLM all the context it needs and let it go ham", and the other saying "Carefully guide and craft it and review everything". Both subthreads had people agreeing.

My experience with LLMs is leaving them unattended gives very poor results within a handful of iterations, and that carefully guiding them can be a value add, but the effort ot do that is very often as much as just writing the damn code myself, particularly if there's a few iterations that go on.

I think the main cause of different views is just the fact that people have very different products and codebases they're working on. Therefore, the requirements and applicability of AI varies a lot, case-by-case. It would be useful if people always explained their exact use cases when making claims one way or another. I understand it's not feasible for many use cases to reveal exact details, but at least some transparency would be great.

I've found AI to be immensely useful in gamedev, mainly UX development, research, and curation. In those areas I let it go quite wild, with mechanical and visual tests for UX, and summaries for research and curation. I've also made skills, custom subagents, hooks, and scripts (all made with AI assistance) to streamline development.

Gamedev is a good example of where AI shines, since the risks of fucking things up are not so big. If I worked on, let's say banking, I'd be much more careful. On my day job, as a data scientist (moving towards product owner) in a large industrial company, I'm much more careful. UX can run on vibes, but anything that touches data processing or business logic needs to be looked at by a human, row by row. Depending on use cases, it may be fine to look at only the tests. The main problem is that there's so much stuff to review, and maybe 98% of it is fine. It's then very boring to try to spot the 2%. I don't have good solutions for that yet, and idk if anyone does.

I've found that these models are extremely good at reasoning since late last year, but they're basically 200 IQ Amelia Bedelias. So my working model earlier this year was to give it a brief instruction and watch how it failed, which was generally a lack of context that it couldn't have possibly known. Then I'd add that context to the team AI docs repo, try again from scratch, and iterate until it did what I wanted without me having to give a bunch of detail in the prompt. Over the course of 2-3 months, I needed to give new context and steering less and less.

I also encouraged my team to paste any inquiries or alerts we got in our slack channel verbatim into codex and follow that procedure until it could figure it out as a one-shot.

I still treat it as more of a design and coding partner day-to-day, so I'm typically not trying to one-shot, but it can e.g. be triggered by a pagerduty alert, reference our code, check grafana panels, query application state across different servers/clusters, and come up with bug fix PRs all autonomously, and then post its analysis, mitigation suggestions, and PR link to slack for the on-caller to review. This was all just some docs, scripts, and a little listener service to trigger a one-shot model prompt that we built.

As a design partner, it's made it easier to add more telemetry or create rapid prototypes to check my assumptions. I've encouraged my team to regularly ponder on what facts they wish they knew, no matter how difficult it would be to find out. Pretend you could just ask God anything you want to know. Then ask the AI to go add whatever's needed to find out. We should all move toward a perfect intuition for what our system is doing because all questions can now be easily answered if we can only think to ask them.

It’s a very soft skill that changes as the capabilities of AI change.

But one thing that doesn’t change is the need to specify an end goal correctly and precisely. I think the emphasis of knowledge/information-processing work is going to increasingly be placed on verification mechanisms. This is practically equivalent to precisely defining an end goal.

Spend time deeply thinking about what it means for a solution to be correct. What properties will a correct solution have? What of those properties are testable? Write those things down and tell the agent.

As models get better, agents will be able to target more and more difficult end goals. The strategy just becomes more useful. (It’s useful for people as well.)

For example, if you want an agent to write a photo editor, think about what end properties the editor should have. There are reference images for color space and rendering transformations. That’s a good start.

Sometimes the goal will be fuzzier. “I want a feature like CaptureOne where I provide a reference image and it makes my image look like that.” Well, time to think really hard about what that means. Iterate with AI on how to test for that precisely. Come up with some good metrics/heuristics. Maybe it means local contrast should match. Maybe the overall distribution of colors. Maybe something more complicated.

Then you have a target and you can let an implementation agent work against that. If it fails, it’s either because the agent is bad or your target was incorrect or incomplete. As models get better, the limiting factor becomes your ability to correctly define a problem.

The reality of the current situation is that we have dozens of families of different model; each of which is differentiated further into various revision numbers and power levels, and depending on which one you use and what the nature of the tasks is, you very much need a different usage style that can only be discerned after multiple (Sometimes very many) sessions of using that individual model for that category of task. It is not one size fits all, and trying to pretend that it is will lead to failure.

Latest models like Opus 5.0 work well in both scenarios. You can just give it a goal and let it make a plan, review the plan and then let it go ham.

Or you can also use it piece-meal and have it do parts of work that you want. It works in both ways and works really well.

I don't think there is any model right now that you can use with 0 oversight, but you can do pretty complex stuff without writing single line of code at this point.

I don't generally use AI to write code I feel I can write efficiently. In these cases, I'll just ask AI for a review. But when I need to write code in areas I'm less confident about, I'll let AI take the reins and the do a thorough review myself. These instances are not too common, though.

I also ask direct questions to AI for brainstorming and advanced programming language usage. In these cases, AI isn't touching the code.

What can be concluded is that some get immense value out of Ai and some are incredibly critical of that claim.

I do not know what the secret sauce is.

But I have a responsibility to the team I lead. I need to protect our velocity. Which means that my hiring practices now include specific checks that people are able to work with ai.

In the end, it might be a step thing. Some people are just inherently incapable of working with this technology...

The truth is that using AI well is a skill much like anything else. There's no one single way to use it much like there's no one single way to write a program that serves hackernews.