Why even bother with the GPU at that point? CPU would be just as fast if you're bottlenecked on SSD bandwidth.

CPU is much slower than GPU. You can actually use both by offloading some layers to CPU as o.offload_layers_to_cpu(layers_num=12). It is faster to load from RAM than from SSD.