If you’re building from source, the vulkan backend is the easiest to build and use for GPU offloading.

Yes, that's what I tried first. Same issue with trying to allocate more memory than was available.