| |
| ▲ | augment_me a day ago | parent [-] | | What a non-answer. And there should be perfect latency hiding, and there should always be predictable power-of-2 workloads, and all weights should always be homogenous, etc. | | |
| ▲ | adgjlsfhk1 a day ago | parent [-] | | It's a reasonable answer. Consumer chips are idle 95% of the time, so idle power draw matters. AI datacenters are currently under capacity, so rather than optimizing for idle power draw, just have work to do. | | |
| ▲ | augment_me a day ago | parent [-] | | No it's not, it does not reflect reality. I have worked with both industry and public clusters. The average is around 50% idle-time, many university clusters sit at around 80% idle-time. The absolute best I have seen at a hyperscalar internal with predictable, constant workloads due to own projects, and cracked routing infra is 15% idle-time. | | |
| ▲ | Tuna-Fish 21 hours ago | parent | next [-] | | There are both latency-sensitive (serving tokens to users) and non-latency sensitive (doing RLAIF requests to help train models) workloads, and there are enough of the latter chasing compute that it should not be hard to get near perfect utilization. The non-latency sensitive workloads do pay a lot worse. This might become a problem later, but right now, if you have AI compute idling, you are doing something very wrong. | | |
| ▲ | nextaccountic 4 hours ago | parent [-] | | Can you train on Cerebras hardware though? I was under impression it's for inference only | | |
| ▲ | adgjlsfhk1 2 hours ago | parent [-] | | a substantial portion of training the most recent couple gens of AI is inference (through reinforcement learning). |
|
| |
| ▲ | imtringued 16 hours ago | parent | prev [-] | | Even if they can't "idle" they can still turn off whole nodes. I'm not really sure what your point is. |
|
|
|
|