Wouldn't a layer per MCU be heavily bandwidth constrained?

Depending on where you slice the model up, it can be not a whole lot of data. For instance each transformer block outputs a single vector in an embedding space.

I can see that being cheaper to bitbang with PIO than to actually compute.

There's certainly some latency stack up, but throughput should be remarkably good.