Yep - this has worked well for me too. I do it a little differently:

I have a /review-sessions command & a "parse-sessions" skill that tells Claude how to parse the session logs from ~/.claude/projects/, then it classifies the issues and proposes new skills, changes to CLAUDE.md, etc. based on what common issues it saw.

I've tried something similar to DISCOVERIES.md (a structured "knowledge base" of assumptions that were proven wrong, things that were tried, etc.) but haven't had luck keeping this from getting filled with obvious things (that the code itself describes) or slightly-incorrect things, or just too large in general.

When you get stuck in a loop it's best to remove all code back to a point it didn't have problems. If you continue debugging in that hammering failure loop you get TONS of random future bugs.

Nothing about this feels inflexible to me at the moment - I'm evolving the way I use these tools on a daily basis, constantly discovering new tricks that work.

Just this morning I found out that I can tell Claude Code how to use my shot-scraper CLI tool to debug JavaScript and it will start doing exactly that:

  you can run javascript against the page using:
  shot-scraper javascript /tmp/output.html \
  'document.body.innerHTML.slice(0, 100)'
  - try that

With great love to your comment, this has the same vibes as the infamous 2007 Dropbox comment: https://news.ycombinator.com/item?id=9224

I'd also argue that the context for an agent message is not the commit/release for the codebase on which it was run, but often a commit/release that is yet to be set up. So there's a bit of apples-to-oranges in terms of release tagging for the log/trace.

It's a really interesting problem to solve, because you could in theory try to retroactively find which LLM session, potentially from days prior, matches a commit that just hit a central repository. You could automatically connect the LLM session to the PR that incorporated the resulting code.

Though, might this discourage developers from openly iterating with their LLM agent, if there's a panopticon around their whole back-and-forth with the agent?

Someone can, and should, create a plug-and-play system here with the right permission model that empowers everyone, including the Programmer-Archaeologists (to borrow shamelessly from Vernor Vinge) who are brought in to "un-vibe the vibe code" and benefit from understanding the context and evolution.

But I don't think that "just dump it in clickhouse" is a viable solution for most folks out there, even if they have the infrastructure and experience with OTel stacks.

I'm currently experimenting with trying to do this through documentation and project planning. Two core practices I use are a docs/roadmap/ directory with an ordered list of milestone documents and a /docs/retros/ directory with dated retrospectives for each session. I'm considering adding architectural decision records as a dedicated space for documenting how things evolve. The quote fta could be handled by the ADR records if they included notes on alternatives that were tried and why they didn't work as part of the justification for the decision that was made.

The trouble with this quickly becomes finding the right ones to include in the current working session. For milestones and retros it's simple: include the current milestone and the last X retros that are relevant but even then you may sometimes want specific information from older retros. With ADR documents you'd have to find the relevant ones somehow and the same goes for any other additional documentation that gets added.

There is clearly a need for some standardization and learning which techniques work best as well as potential for building a system that makes it easy for both you and the LLM to find the correct information for the current task.

> agentic capabilities are very much on a roll-your-own-in-elisp basis

I use gptel-agent[1] when I want agentic capabilities. It includes tools and supports sub-agents, but I haven't added support for Claude skills folders yet. Rolling back the chat is trivial (just move up or modify the chat buffer), rolling back changes to files needs some work.

[1] https://github.com/karthink/gptel-agent

Running "rg" in your ~/.claude/ directory is a good starting point, but it's pretty inconvenient without a nicer UI for viewing the results.

LLMs notoriously don't learn anything - they reset to a blank slate every time you start a new conversation.

If you want them to learn you have to actively set them up to do that. The simplest mechanism is to use a coding agent tool like Claude Code and frequently remind it to make notes for itself, or to look at its own commit history, or to search for examples in the codebase that is available to it.

If by "these things" you mean large language models: they are not learning. Famously so, that's part of the problem.

No, we’re the ones who are learning.

There’s some utility to instructing them to ‘remember’ via writing to CLAUDE.md or similar, and instructing them to ‘recall’ by reading what they wrote later.

But they’ll rarely if even do it on their own.