This seems to be running on an FPGA board that costs ~$300? Anyone know more about the hardware?

Its a datacenter decommissioned board, really popular among hobbyists.

For a TPU focused on inference the name of the game is memory bandwidth. How much of the available bandwidth you can extract for as little logic/area/power as you can.