> IDK how the custom hardware exploits this; would love to hear any ideas!

You might like this article [1], titled "FPGA-based CNN Acceleration using Pattern-Aware Pruning". More context and details can be found in the PhD thesis of Léo Pradels [2].

[1]: https://inria.hal.science/hal-04689673/document

[2]: https://theses.hal.science/tel-05021575v1/file/PRADELS_Leo.p...