Remix.run Logo
jakozaur 2 hours ago

This isn’t limited to large system prompts. Coding-agent harnesses are also becoming more aggressive about using tools, even for trivial requests. In our tests, prompts such as “Hey” or “commit” sometimes triggered 30+ tool calls:

https://quesma.com/blog/the-true-cost-of-saying-hi-to-an-ai-...

Tokenflation seems very real: the number of tokens consumed by simple tasks keeps increasing.

prymitive 2 hours ago | parent | next [-]

I often find myself annoyed when Opus fixes a typo in a comment and decides to run tests, lints and whenever else it can find to run. Often it will start by stashing current changes just to preemptively check if all tests were passing before. And I can blame myself a bit because my rules do say: verify all changes with tests. But as there is that I in AI that is hyped which you’d think means it knows not to put tomatoes into fruit salad …

mh- 2 hours ago | parent | next [-]

> [..] my rules do say: verify all changes with tests

I am a bit surprised that you're disappointed that it does exactly what you told it to - people usually have the opposite complaint.

If you're using it interactively and watching what it changes, I'd trigger the tests when you think it's needed. And if you want to go more hands-off, why not add try to encode the same nuance you'd use into the rule?

redox99 2 hours ago | parent | prev | next [-]

Following rules like "verify all changes with tests" down to a tee is usually a desirable trait in LLMs. Personally I'd leave that behavior there (just like with humans for some tasks like aviation you have them go through checklists even if some stuff you can infer is not needed). But otherwise just make it "always run tests unless you're absolutely sure they can be skipped".

dymk an hour ago | parent | prev [-]

Add "... unless the changes are trivial, docs-only, or typo fixes" to the "always verify with tests" instruction and see how that does

Chris2048 2 hours ago | parent | prev [-]

> prompts such as “Hey” or “commit” sometimes triggered 30+ tool calls

I read that this is because it wastes time looking through past conversations and other context to figure one what you might want it to do - a less ambiguous prompt would be better.