If you want them to not make shit up, you have to load up the context with exactly the docs and code references that the request needs. This is not a trivial process and ime it can take just as long as doing stuff manually a lot of the time, but tools are improving to aid this process and if the immediate context contains everything the model needs it won't hallucinate any worse than I do when I manually enter code (but when I do it, I call it a typo)

there is a learning curve, it reminds me of learning to use Google a long time ago

▲

th0ma5 6 days ago | parent [-]

So, I've done this, I've pasted in the headers and pleaded with it to not imagine ABIs that don't exist, and multiple models just want to make it work however they can. People shouldn't be so quick to reply like this, many people have tried all this advice... It also doesn't help that there is no independent test that can describe these issues, so all there is anecdote to use a different vendor or that the person must be doing something wrong? How can we talk about these things with these rhetorical reflexes?

▲

verdverm 6 days ago | parent [-]

There is a significant gap between agents and models.

Agents use multiple models, can interact with the environment, and take many steps. You can get them to reflect on what they have done and what they need to do to continue, without intervention. One of the more important things they can do is understand their environment, the libraries and versions in use, fetch or read the docs, and then base their edits on those. Much of the hallucinating SDKs can be removed with this, and with running compile to validate, they get even better.

Models typically operate in a turn-by-turn basis with only the context and messages the user provides.

▲

th0ma5 5 days ago | parent [-]

You can't make any guarantees and manually watching everything is not tenable. "Much" instead of "all" means having to check it all because "much" is random.

	▲	verdverm 5 days ago \| parent [-]
		You don't have to watch it like you don't have to watch your peers. We have code review processes in place already You're never going to get all, you don't have all today. Humans make mistakes too and have to run programs to discover their errors