Apple needs on-device AI to do chores for me with the apps I have installed. Apple has everything it needs:

* Apps are already logged in, so no extra friction to grant access.

* Apps mostly use Apple-developed UI frameworks, so Apple could turn them into AI-readable representations, instead of raw pixels. In the same way a browser can give the AI the accessibility DOM, Apple could give AIs an easier representation to read and manipulate.

* iPhones already have specialized hardware for AI acceleration.

I want to be able to tell my phone to a) summarize my finances across all the apps I have b) give me a list of new articles of a certain topic from my magazine/news apps c) combine internet search with on-device files to generate personal reports.

All this is possible, but Apple doesn't care to do this. The path not taken is invisible, and no one will criticize them for squandering this opportunity. That's a more subtle drawback with only having two phone operating systems.

> iPhones already have specialized hardware for AI acceleration.

This really is the problem. Why do I spend hundreds of dollars more for specialized hardware that’s better than last years specialized hardware if all the AI features are going to be an API call to chatGPT? I am pretty sure I don’t need all of that hardware to watch YouTube videos or scroll Instagram/web, which is what 95% of the users do.

> Apple needs on-device AI to do chores for me with the apps I have installed

Nevermind that—iOS just needs to reliably be able to play the song I’m telling it to without complaining “sorry, something went wrong with the connection…”

I agree completely, it's really unfortunate how AI on apple devices has been going. The message summarization is borderline useless and widely mocked, meanwhile their giant billboard ads for it are largely stupid and uncompelling. Let me choose to give it access to my data if I want to do really useful stuff with on device processing. They've been leaning into the privacy thing, do the stuff that would be creepy if it left my device, generate push notification reminders for stuff I forgot to put in the calendar, or track my location and tell me I'm going to the wrong airport. Suggest birthday gifts for my friends and family, idk.

Edit: And add strong controls to limit what it can and cannot access, especially for the creepy stuff.

I think on-device AI will show up more front and center but in a few more years.

A big issue to solve is battery life. Right now there's already a lot that goes on at night while the user sleeps with their phone plugged in. This helps to preserve battery life because you can run intensive tasks while hooked up to a power source.

If apps are doing a lot of AI stuff in the course of regular interaction, that could drain the battery fairly quickly.

Amazingly, I think the memory footprint of the phones will also need to get quite a bit larger to really support the big uses cases and workflows. (I do feel somewhat crazy that it is already possible to purchase an iPhone with 1TB of storage and 8GB of RAM).

> In the same way a browser can give the AI the accessibility DOM, Apple could give AIs an easier representation to read and manipulate.

Apps already have such an accessibility tree; it's used for VoiceOver and you can use it to write UI unit tests. (If you haven't tested your own app with VoiceOver, you should.)

This is all possible, but an absolutely terrible idea from a security point of view, while prompt injection attacks are still a thing, and there's little evidence they will stop being a thing soon.

They've being doing some research on this: https://machinelearning.apple.com/research/ferretui-mobile

Apple is generally anti market hype. It is a smart PR move to avoid mentioning AI after the Apple Intelligence fiasco, their researchers leaving, and the bubble sentiment at the moment.

[dead]