I MVP'd one of these (a basic sequence of LLM customer support 'agents') at my last job, I guess spring 2024. So much has changed since then!

'Routing through increasingly specialised agents' was my approach, and the only thing that would've done the job (in MVP form) at the time. There weren't many models that would fit our (v good) CS & Product teams' dataset of "probable queries from customers" into a single context window.

I never personally got my MVP beyond sitting with it beside the customer support inbox, talking to customers. And AFAIK it never moved beyond that after I left.

Nor should it have been, probably - there are (wild, & mostly ineffable) trade-offs that you make the moment you stop actually talking to users at the very moment they get in touch. I don't remember ever making a trade-off like that where it was worthwhile.

I _do_ remember it as perhaps the most worthwhile time I ever spent doing product-y work.

I say that because: To consider a customer support query type that might be 0.005% of all queries received by the CS team, even my trash MVP had to walk a path down a pretty intricate tree of agents and possible query types.

So - if you believe that 'solving the problems users have with your product' = 'making a better product'. then talking to an LLM that was an advocate for a tiny subset of users, and knew very intimately the details of their issue with your product, that felt really good. It felt like it was a very pure version of what _I_ should be to devs, as any kind of interface between them and our users.

It was very hard to stay a believer in the idea of a 'PM' after seeing that, at least. As a person who preferred to just let people get on with things.

I enjoyed the linked post; it's really interesting to see how far things have come. I'm surprised nobody has built 'talk to your customers at scale', yet - this feels like a far more interesting problem than 'avoid talking to your customers at scale'.

I'm also not surprised, I guess, since it's an incredibly bespoke job to do properly, I imagine, for most products.

> I enjoyed the linked post; it's really interesting to see how far things have come. I'm surprised nobody has built 'talk to your customers at scale', yet - this feels like a far more interesting problem than 'avoid talking to your customers at scale'.

This sounds hard to pull off in a very similar way to getting good data through surveys.

I generally don't want to talk to my tools. If I'm motivated to talk to you, it's probably because something went wrong. And even if I talked to you when not annoyed, I'd struggle to articulate more than "it's working good" at any given moment - when what you really want as a product person is to know "it's working good, but I had to internalize this workaround for something for my use case that no I don't even think about but originally I found offputting and almost bounced because of" or whatever.