In our new world of non-deterministic output (that's why we love LLMs! they say such helpful/agreeable/sometimes wrong stuff!), I think CI won't be sufficient. CI is in the realm of Quality Control; when I build the thing, is it to spec and does it do what I need it to do?

But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.

In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time. For example, if a hotel chatbot starts telling guests checkout time is at 11pm instead of 11am, that's a real operational problem and ideally should be caught before it impacts customers.

LLMs introduce new classes of problems/risks that we are just starting to understand and develop the tools to manage.

> In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time.

Is that hotel example real? Curious how exactly you employ this technique—my naive idea was, if talking software development, a sort of 'sanity-check auto-linter agent' catch errors on a regular basis (every 10 seconds, every write, w/e).

>But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.

The instinct seems good because it's impossible to keep up with all the details if you are running AI full-blast. Absolutely impossible. So testing outputs makes sense.

I have a hard time seeing exactly how we get from here to there. But intuitively I would not be surprised. One of these thing where quality may drop 20% but you can scale 100x.