On a related note, this type of reasoning is what made me flip my opinion on microservices. I've generally been skeptical of a many-microservice architecture for the last decade but LLMs change that - a small microservice is more likely to fit in a context window.

I think this gestures at a more general point - we're still focusing on how to integrate LLMs into existing dev tooling paradigms. We squeeze LLMs into IDEs for human dev ergonomics but we should start thinking about LLM dev ergonomics - what idioms and design patterns make software development easiest for AIs?

They don't need to be services. You can - and many projects do - structure your code as a set of loosely coupled modules. Each module has a responsibility or set of responsibilities. They communicate with each other via well defined interfaces. For exposing code like this to an LLM, you would have them make a change to one or sometimes two modules, with access to the interface docs of all the other modules. The disadvantage of this compared to microservices is that if a module crashes it will take the entire process down with it, you can't move a module onto a different machine or create multiple instances of it as easily, etc. The advantage is that communication is done via function calls, which are simpler and more efficient than rpc.

> I think this gestures at a more general point - we're still focusing on how to integrate LLMs into existing dev tooling paradigms.

This is what we should be doing. This for a couple reasons. For one thing, humans don't have an entire codebase "in context" at a time. We should be recognizing that the limitations of an AI mirror the limitations of a person, and hence can have similar solutions. For another, the limitations of today's LLMs will not be the limitations of tomorrow's LLMs. Redesigning our code to suit today's limitations will only cause us trouble down the road.

Microservices are about deployment, less about code structure. You can have the same code modularization like microservices provide within a monolith instead, for example in the form of libraries. Conversely, you can in principle build several distinct microservices out of the same shared codebase.

[dead]