Another interesting potential market here will be 'LLM in a box'. All the hardware and other tooling in a prebuilt, but modular, package ready to go. Pay one up-front cost, get a system running [whatever open LLM] with a token rate of [x], optionally configured to be immediately ready for distributed usage. Basically the opposite of cloud stuff: no rent, no dependency, 100% guaranteed uptime, guaranteed security/privacy (at least subject to your own actions), and so on.

▲

adrian_b an hour ago | parent | next [-]

Palantir already offers a "turnkey AI datacenter", i.e. a rack with "NVIDIA Blackwell Ultra systems with eight NVIDIA Blackwell Ultra GPUs and NVIDIA Spectrum-X™ Ethernet networking for AI training and inference".

It is said that it comes with all hardware and software required to run inference or training with an open weights LLM.

The existence of this product, which competes with cloud-based offerings like those of OpenAI and Anthropic, is presumably the reason why the Palantir CEO criticized very harshly some time ago the business model of OpenAI/Anthropic.

While I doubt that the ethics of Palantir is any better than of OpenAI/Anthropic, in this particular case I have to agree with Alex Karp about "Sovereign AI", i.e. that only losers will make their business completely dependent on an external entity like OpenAI or Anthropic, who are certainly not trustworthy.

▲

vrganj an hour ago | parent [-]

I'm not sure a data center run by ... Palantir of all organizations is what people have in mind when they worry about data sovereignty.

	▲	adrian_b an hour ago \| parent [-]
		They are selling it, not running it. It is just a dedicated computer system, which should be managed by its owner, like any other on-prem servers. I doubt that it has a good price/performance ratio, but it is a solution for those who feel that they do not want to search, buy, assemble, install and configure every HW/SW component.

▲

jurgenburgen 2 hours ago | parent | prev | next [-]

> 100% guaranteed uptime

Disagree there but I think this is an interesting idea. We would need to find some more cost-efficient hardware to run it on than Nvidia GPUs.

	▲	pulse7 an hour ago \| parent [-]
		It will come... all big hardware players (Intel, AMD, Broadcom) and dozens of startups (Tenstorrent, etc.) are working on it...

▲

bevekspldnw an hour ago | parent | prev | next [-]

“100% guaranteed downtime when you least can afford it and the support tickets are your problem.”

We’ve a hybrid shop, including hosting our own ML infra, and we save a ton from cloud spend with local ML. Easily one million USD over past three years. But it’s not “free”, you are shifting a lot of labor into your plate.

▲

hypfer an hour ago | parent [-]

And with that also gain institutional knowledge, skill up your workers and attract talent that wants to work on this stuff.

All boils down to short-term/long-term thinking.

	▲	gruturo 21 minutes ago \| parent [-]
		This. People WANT to work on this stuff. And having skilled workers is a precious advantage.

▲

Godsend69 an hour ago | parent | prev [-]

[dead]