My agent, whenever changes are made, happily keeps those .md files up-to-date with newest changes.

Mine doesn't. It mostly adds and keeps referencing previous decisions. Correctly, referring to them as being overridden. Even tweaking the original text or occasionally deleting a part. But text would look differently had it been written from scratch.

Unless a LLM works truly as a compiler, there will be a drift between the code and the spec.

For me they mostly append the end of the file with more sections or details. Heck it often adds very specific implementation details that it worked on recently that have no business being written in a repo wide summary.