I still think NVIDIA is a bad bet--where is their moat in the long term? Doesn't the sort of work NVIDIA engineers do look vulnerable to AI-assisted automation? NVIDIA engineers code against a well-defined test suite/specification, right?

▲

ismailmaj 2 hours ago | parent | next [-]

Their moat is cuda and cuda libraries and everything built on top.

When a new architecture drops, it's always PyTorch running on CUDA, other PyTorch backends are best effort, even if they reach feature parity, many industry power users went closer to the metal to squeeze performance and that stuff is too specific to Nvidia stuff.

if there is something that will beat Nvidia, it won't be something reaching feature parity with slightly better economics (like AMD, also Nvidia could just reduce their margins), it needs to be a novel approach worth rewriting the codebase for (maybe Cerebras, maybe a new player).

▲

0xDEAFBEAD an hour ago | parent [-]

I don't understand why AMD can't offer a drop-in replacement for cuda which implements an identical API.

How much actual diversity is there among standard AI workloads? I would expect this is an 80/20 thing where 80% of the workload uses 20% of the features.

>Nvidia could just reduce their margins

Commoditization is great for stock prices ;-)

	▲	ismailmaj 26 minutes ago \| parent [-]
		3 things, they can, there is a precedent for that with Google v. Oracle for Java, and they have something! AMD engineered something called HIP which is CUDA API compatible libraries that targets AMD's hardware, it's the closest thing we have for drop-in replacement to Nvidia's software moat. It works for simple stuff but loses terribly for frontier kernels (like Flash Attention 3), novel approaches (e.g. Mamba) or networking (e.g. NCCL), also they are rough on the edges, so what you gain from GPU costs is lost in engineering cost. My previous company tried to compete in this GPU game while putting effort to have a good software stack (Rivos), drop in replacement and cheaper with decent software. But that vision was rough, any new player had to implement the bad APIs due to backward compatibility concerns, following specs wasn't sufficient as a lot of the AI stack was depending on observable effects (Hyrum's Law), and Nvidia simply just had a long head start, the company is now dead (acquired by Meta) and AFAIK there isn't another player. Best case scenario AMD puts more effort into their software stack but I just think they do not have enough internal talent to compete. Training will continue to be an Nvidia's thing and that's where most of the money sits, unless suddenly the AI research scene pivots to using JAX but I do not see it coming any time soon, if anything, I've seen internal efforts at Google to make PyTorch work nicely with TPUs. Some players like Anthropic started using JAX for training but all the small players are using Nvidia, I'm guessing it has something to do with Nvidia partnering aggressively with startups.

▲

pjc50 4 hours ago | parent | prev [-]

AMD should have been ideally placed to compete with them, and haven't.

> NVIDIA engineers code against a well-defined test suite/specification, right?

The spec is the value. And the patents.

	▲	0xDEAFBEAD 3 hours ago \| parent [-]
		I admit I'm not too knowledgeable about the semiconductor industry. But it seems to me that there two likely scenarios: AI Bear or AI Bull. In the AI Bear scenario, NVIDIA is obviously overvalued. In the AI Bull scenario, we get full automation of software engineering. With "just a few clicks", an AMD employee can extract and replicate whatever subset of the spec is needed for AI workloads. Didn't the Google vs Oracle case find that copying an API can be fair use? And NVIDIA's patents haven't stopped Google from training on TPUs have they?