Depends what you use perhaps. I use codex and it seems to mostly stick to instructions I give. I use an AGENTS.md that explicitly points to the repository's skill directory. I mostly keep instructions in there for obvious things like how to build, how to test, what to do before declaring a thing done, etc. I don't tend to have a lot of skills in there either.

Probably the more skills you have, the more confused it might get. The more potentially conflicting instructions you give the harder it gets for an LLM to figure out what you actually want to happen.

If I catch it going off script, I often interrupt it and tell it what to do and update the relevant skill. Seems to work pretty good. Keeping things simple seems to work.