| ▲ | rahimnathwani 4 hours ago | |
Has anyone successfully run this on a Mac? The installation instructions appear to assume an NVIDIA GPU (CUDA, FlashAttention), and I’m not sure whether it works with PyTorch’s Metal/MPS backend. | ||
| ▲ | magicalhippo 2 hours ago | parent | next [-] | |
FWIW you can run the demo without FlashAttention using --no-flash-attn command-line parameter, I do that since I'm on Windows and haven't gotten FlashAttention2 to work. | ||
| ▲ | javier123454321 4 hours ago | parent | prev | next [-] | |
I recommend using modal for renting the metal. | ||
| ▲ | turnsout 2 hours ago | parent | prev [-] | |
It seems to depend on FlashAttention, so the short answer is no. Hopefully someone does the work of porting the inference code over! | ||