So doing it on a big GPU would be pretty helpful. I wonder if anyone has used CUDA for this, or if any hardware synths have GPUs.

The time it takes to get data to/from the GPU is problematic given the sample rate you need to deliver.

Anukari used the GPU but eventually abandoned it for the CPU. They have some good and informative blog posts about it.