My conversation

"Design a schema like Calendly" --> Did it

"OK let's scale this to 100m users" --> Tells me how it would. No schema change.

"Did you update the schema?" --> Updates the schema, tells me what it did.

We've been running into this EXACT failure mode with current models, and it's so irritating. Our agent plans migrations, so it's code-adjacent, but the output is a structured plan (basically: tasks, which are prompt + regex. What to do; where to do it.)

The agent really wants to talk to you about it. Claude wants to write code about it. None of the models want to communicate with the user primarily through tool use, even when (as I'm sure ChartDB is) HEAVILY prompted to do so.

I think there's still a lot of value there, but it's a bummer that we as users are going to have to remind all LLMs for a little bit to do keep using their tools beyond the 1st prompt.

I asked it to abstract a event-specific table to a GP "events" table which it did, but kept the specific table. I asked it to delete that table and it said it did, but did not. I got stuck in a loop asking it to remove the table that the LLM insisted was not part of the schema, but was present in the diagram.

It was easier to close the tab than fire a human, but other than that not a great experience.

isnt this what the agents are for, you assign them jobs to make changes then evaluate those changes. there is a necessary orchestration piece and maybe even a triage role to sort through things to do and errors to fix