>local models run on your mac for your workflow.

10 years ago 32GB ram laptops sounded too much. 8 was enough. These days even I would get that much ram since it’s soldered. 64GB is higher end.

In a few years we should see such high end hardware commonplace. Working with a local LLM to get work done is the ideal way to go which has mostly hardware limitation as of now that gets solved in due time.

>I have found it pretty amazing that Apple basically didn't do much with AI software

The iPhone 15 was almost entirely marketed based upon AI (I would say fraudulently so, advertising features they still haven't delivered), and a huge portion of the OS work was on local AI or AI integration.

And for that matter Apple has been dumping enormous sums into their own AI development. Their failure to have a lot to show for it doesn't void the fact that they tried really, really hard.

It's bizarre how often this "Apple sat on the sidelines and let the AI people fight...so smart!" narrative appears on HN. Apple hasn't gone down the path of spending hundreds of billions on nvidia GPU data centres, but they absolutely tried really hard to matter in AI.

|It's possible that they're working on their own LLM

Yep, Siri AI; they’re doing it in public.

nth mover advantage.

The real downside for me is not having Linux support.

It would take Apple one or two engineers to make Linux life much easier on macs. But Linux is outside their walled garden so it's ignored.

It's worth noting that the 5090 (or the RTX Pro 6000 big brother with 92GB VRAM) will run rings around the Mac when it comes to compute.

My old 3090 is typically significantly faster (almost 2x token/s) than my M4 Max 128GB machine, as long as the model fits in the 24GB of VRAM.

In most situations it's a better idea to just buy tokens. But there are definitely cases when that's not an option. And then a machine like the M5 Ultra can allow you to do things locally for a fairly limited budget. And in a simpler package to manage than a machine with multiple GPUs.

is it still effectively 2/3rds? Don't know enough to compare a discrete GPU/CPU setup to something like this where it's more integrated

How much memory does that come with?

I suspected Apple would let everyone else blow all their money then, when the dust settles, deliver a better experience to end users and clean up.

I love linux and would be using it if the ARM support was better. It's just not there and most distros that support ARM do it a little poorly. I just haven't seen anything even remotely comparable to Apple Silicon and unfortunately Linux is struggling very hard to support it.

You can't run Linux directly on these. Asahi Linux supports up to M2 only.

Linux runs very well in a VM on macOS. There are many good options for this, some free and open source (QEMU, UTM, Lima, Colima), some proprietary (VMware Fusion, Parallels).

But Linux in a VM doesn't get access to the real GPU, so model performance is limited. Those running on the CPU perform well, and those needing the GPU don't.

However, macOS on M-series macs is excellent for local models. (Maybe not as excellent as a box full of the best nVidia GPUs, but still excellent).

So if you're getting Apple hardware, like Linux, and want to run all of it locally, a fine setup for a machine to run local models, with agentic characteristics:

- macOS running one of the many local model runners. I used to use Ollama and Whisper, and now use llama.cpp instead of Ollama. Others use LM Studio, oMLX, etc. Provide HTTP endpoints to access the models.

- Linux in a VM for overall control and orchestration, with standard VM settings, and bridged networking so it appears as its own machine on your network. Also, in here provide a robust shared file server for shared state. Use this VM as your desktop and primary access to the machine, if you like Linux.

- Linux in a VM to launch ephemeral, volatile containers, with the containers using a memory-only tmpfs overlay on top of a read-only Linux filesystem in a VM disk image, with tools in this filesystem. Alternatively, a writable Linux filesystem in a VM disk image, with disk buffering set to use macOS host buffering and discard fsync requests. These settings optimise for container disk performance for data that's only ephemeral which will be deleted soon or on system shutdown. (You can combined both VMs, but need to use two VM disks to get equivalent behaviour, and be careful about VM disk configuration of the two disks.)

- Containers spawned within that second Linux VM can be spawned very quickly and run quickly, so are ideal for LLM agents that need a quick sandbox. These sandboxes generally run faster than a macOS sandbox, despite being on the same machine with VM overhead, because Linux is faster at some things. Teach the LLMs to store files and memories they want to keep in the shared file server.

I guess, what I mean is: Why are these tiny aluminum boxes so optimal?

I just want my butt ugly repairable beast machine to do the same trick. Why is my ram not unified? I have an iGPU in my server, but it can't access the 64 GB ram (I got last year for 150 euro) directly or something? It's on the CPU right? Why did only Apple go for this architecture? So many questions...

Strix platform maybe?

Asahi was stuck at M3 last time I checked it out.

AFAIK, apple does not release drivers open source, asahi is a reverse-engineering endeavour and does not support GPU. For nvidia, there are both proprietary and open-source linux drivers. CUDA and inference works on linux with nvidia. I would recommend checking out this video of Alex Ziskind to shop for a computer to run local LLMs: https://www.youtube.com/watch?v=mevUEQcumzU&t=224s. TL;DR besides Apple he recommends, DGX Spark, Tenstorrent Wormhole N300, AMD Radeon 7900 and NVIDIA RTX 5090.