I have seen it doing incredible stuff. One shotted adding a feature that included modifications to a proprietary backoffice system, db schema updates, defining new api models, implementing changes on the backend and then on the frontend.

I've also seen seen it choking when tasked to add a simple result count on a search.

The short answer is, it's cheap to let it try.

Is it cheap? It adds up really quickly. One shot at trying to build an iteration of a simple python app (<1000 LOC tops) can cost between $1 and $5. And that’s a single attempt.

And this is just the tip of the tip of the iceberg of what even a medium sized startup spends. This is not cheap in any way.