I've found a good balance in looking through sessions based on user feedback (ex: user thumbs down and submits it into a review queue) to build an understanding of usage patterns and failure modes. This is very boring but critical to understanding how the LLM is trying to navigate the problem. Then you use your insights to create offline evals. Then hill climb those. Then A/B test the changes with your uses.

I don't know if this is the end state, but I think this is pretty close to what I was seeing a year ago and what we will be seeing next year. Yea, maybe we use agents to do the looking through but you will still be shepherding that.