This is a question that pops up all over the place these days. I can offer one answer.

System design is principally a process of identifying the best possible points at which to decouple the components that will make up the system. A clean design means that components can be swapped out for alternatives easily. Messy designs result in interlocking complexity.

Structuring your components as a general-purpose re-usable library forces you to consider how it is decoupled from consumers. If you do a good job designing it, it should be usable by all sorts of consumers, so you might as well make it a standalone library.

To a first approximation, I think this process of identifying clean decoupling points is the main thing that humans still need to babysit their agents over.