Remix.run Logo
▲ Tuna-Fish 21 hours ago

There are both latency-sensitive (serving tokens to users) and non-latency sensitive (doing RLAIF requests to help train models) workloads, and there are enough of the latter chasing compute that it should not be hard to get near perfect utilization. The non-latency sensitive workloads do pay a lot worse.

This might become a problem later, but right now, if you have AI compute idling, you are doing something very wrong.

▲nextaccountic 4 hours ago | parent [-]

Can you train on Cerebras hardware though? I was under impression it's for inference only

▲adgjlsfhk1 3 hours ago | parent [-]

a substantial portion of training the most recent couple gens of AI is inference (through reinforcement learning).