Which is more important for AI/LLM stuff, the number of GPU cores or the amount of unified memory? I can choose from a Mini with the M5 Pro 18CPU/30GPU and 64GB RAM or a Studio with a M5 Max 18CPU/40GPU and 48GB RAM for roughly the same price

They affect different things. You want more powerful models -> more memory. You want the models that run to be faster -> more gpu cores.

Larger model -> more memory

It depends on what model(s) you want to run.

Memory bandwidth is one of the most important factors. The Max machines typically have higher bandwidth than Pro (you should look up these exact models to verify though). Having said that, you could run bigger models with 64GB.

I went with a 64GB M2 Max a few years back even though it stretched my budget and I'm glad I did. I'm not sure what I'd pick in this case. Tough call.