I use it on a 24gb gpu Tesla P40. Very happy with the result.

hkt 4 days ago | parent [-]

Out of interest, roughly how many tokens per second do you get on that?

Like 4. Definitely single digit. The P40s are slow af

	▲	coolspot 4 days ago \| parent [-]
		P40 has memory bandwidth of 346GB/s which means it should be able to do around 14+ t/s running a 24 GB model+context.