This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!

▲

Lerc 5 hours ago | parent [-]

One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions.

PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton of them running all at the same time.

The large scale dedicated hardware systems will still have the edge for performance per watt, but the low entry level and slow incline does make these things quite appealing.

▲

hgoel 2 hours ago | parent | next [-]

Wouldn't a layer per MCU be heavily bandwidth constrained?

	▲	monocasa 2 hours ago \| parent [-]
		Depending on where you slice the model up, it can be not a whole lot of data. For instance each transformer block outputs a single vector in an embedding space. I can see that being cheaper to bitbang with PIO than to actually compute. There's certainly some latency stack up, but throughput should be remarkably good.

▲

NuclearPM 3 hours ago | parent | prev [-]

Run the numbers before you waste time here. I doubt this will work.

	▲	Lerc an hour ago \| parent [-]
		PIO to PIO between two rp2350s should be able to transfer as many bits per clock as you can spare pins for. They have a single cycle double multiply per core, and the interpolators give you a heap of ability The PIO can be awkward, but you can run a bunch of them at once. Going from MCU to MCU you don't even need to involve the CPU cores, PIO to PIO Comms via pins You are obviously not going to get big TOPS from it because a Trillion is a ridiculous amount anyway. But never underestimate the power of controlling the whole pipeline. Ultimately none of the other things I'm doing with MCUs are practical, why would this to be any different.