Remix.run Logo
▲ augment_me a day ago

Very interesting! Thanks for answering with your take, and the link.

Questions:

1) What is the idle power draw of the chip? Is the 80% number wrong?

Nvidia is idling at around 180W/15% of their max for B300. So saying that "it's not unique to Cerebras" is meaningless if we don't consider the power draw itself

2) Decommissioned is not the same as RMA:d/refurbished/redeployed, so giving this number is kind of dodging the question IMO

▲wmf a day ago | parent [-]

None of these expensive chips should be idling.

▲augment_me a day ago | parent [-]

What a non-answer. And there should be perfect latency hiding, and there should always be predictable power-of-2 workloads, and all weights should always be homogenous, etc.

▲adgjlsfhk1 a day ago | parent [-]

It's a reasonable answer. Consumer chips are idle 95% of the time, so idle power draw matters. AI datacenters are currently under capacity, so rather than optimizing for idle power draw, just have work to do.

▲augment_me a day ago | parent [-]

No it's not, it does not reflect reality. I have worked with both industry and public clusters. The average is around 50% idle-time, many university clusters sit at around 80% idle-time.

The absolute best I have seen at a hyperscalar internal with predictable, constant workloads due to own projects, and cracked routing infra is 15% idle-time.

▲Tuna-Fish 21 hours ago | parent | next [-]

There are both latency-sensitive (serving tokens to users) and non-latency sensitive (doing RLAIF requests to help train models) workloads, and there are enough of the latter chasing compute that it should not be hard to get near perfect utilization. The non-latency sensitive workloads do pay a lot worse.

This might become a problem later, but right now, if you have AI compute idling, you are doing something very wrong.

▲nextaccountic 4 hours ago | parent [-]

Can you train on Cerebras hardware though? I was under impression it's for inference only

▲adgjlsfhk1 2 hours ago | parent [-]

a substantial portion of training the most recent couple gens of AI is inference (through reinforcement learning).

▲imtringued 16 hours ago | parent | prev [-]

Even if they can't "idle" they can still turn off whole nodes. I'm not really sure what your point is.