There's a library I use with extensive documentation- every method, parameter, event, configuration option conceivable is documented.

Every so often I get lost in the docs trying to do something that actually isn't supported (the library has some glaring oversights) and I'll search on Google to see if anyone else came up with a similar problem and solution on a forum or something.

Instead of telling me "that isn't supported" the AI overview instead says "here's roughly how you would do it with libraries of this sort" and then it would provide a fictional code sample with actual method names from the documentation, except the comments say the method could do one thing, but when you check the documentation to be sure, it actually does something different.

It's a total crapshoot on any given search whether I'll be saving time or losing it using the AI overview, and I'm cynically assuming that we are entering a new round of the Dark Ages.

I have the Google AI overview adblocked and I keep it up to date because it's an unbelievably hostile thing to have in your information space: it sounds truthy, so even if you try to ignore it it's liable to bias the way you evaluate other answers going forward.

It's also obnoxious on mobile where it takes up the whole first result space.

There's an attempt to kinda have these things documented for AIs, called llms.txt, which are generally hosted on the web.

In theory, an AI should be able to fetch the llms.txt for every library and have an actual authoritative source of documentation for the given library.

This doesn't work that great right now, because not everyone is on board, but if we had llms.txt actually embedded in software libraries...it could be a game changer.

I noticed Claude Code semi regularly will start parsing actual library code in node_modules when it gets stuck. It will start by inventing methods it thinks should exist, then the typescript check step fails, and it searches the web for docs, if that fails it will actually go into the type definition for the library in node_modules and start looking in there. If we had node_modules/<package_name>/llms.txt (or the equivalent for other package managers in other languages) as a standard it could be pretty powerful I think. It could also be handled at the registry level, but I kind of like the idea of it being shipped (and thus easily versioned) in the library itself.

I mean... Yeah I've had ChatGPT tell me you can't do things with Make that you totally can. They aren't perfect. What do you expect Google to do about it?

Yes I know hallucinations are a thing. But when I had problems lile that better prompting (don’t make assumptions) and telling it to verify all of its answers with web resources

For troubleshooting an issue my prompt is usually “I am trying to do debug an issue. I’m going to give you the error message. Ask me questions one by one to help me troubleshoot. Prefer asking clarifying questions to making assumptions”.

Once I started doing that, it’s gotten a lot better.