Gemini Nano is supposedly doing it on device. It looks like something similar should work with Apple GPU and ANE.