| |
| ▲ | verdverm 6 hours ago | parent [-] | | MiMo RL'd their latest models for $3.5M (million, not billion). It's hardware, software, and algos https://mimo.xiaomi.com/rl/ Also look to the history of stable diffusion | | |
| ▲ | _aavaa_ 6 hours ago | parent | next [-] | | I believe that’s post-training, not pre-training. | | |
| ▲ | overfeed 2 hours ago | parent | next [-] | | > I believe that’s post-training, not pre-training Aren't AI companies already butting against the limits of usable training data? Outside of undiscovered techniques, additional costly pre-training will likely yield diminishing returns if you've already scraped the whole internet and destructively scanned as many 20th century books as you could lay your hands on. | |
| ▲ | verdverm 6 hours ago | parent | prev [-] | | yes, just post training, which is the most expensive phase now as I understand it |
| |
| ▲ | lifeisloving 6 hours ago | parent | prev [-] | | Training costs are not in free fall, and not every "post training" process is equal, Everyone does this differently. The costs certainly haven't gone down, to train a new 10T param model from scratch, like at all. You cannot point at Xiaomis tiny model and compare it to training a 10T param model. The levels of dunning krueger are getting out of hand. You also arent calculating the salaries of the people involved, of which there's only a few thousand worldwide in your hiring pool. | | |
| ▲ | overfeed an hour ago | parent | next [-] | | > You also arent calculating the salaries of the people involved, of which there's only a few thousand worldwide in your hiring pool. There were probably a few dozen 20 years ago, and the existing pool grew from somewhere. AI (nee machine learning) is not rocket science - but it is a science, so it rewards those who are literate, curious, can hypothesize, experiment and learn from results to gain an intuition. I bet most statistics-adjacent professionals (millions, worldwide) can move laterally into AI research and excel at it. Nevermind AI has been among the most-in-demand undergrad programs since ChatGPT dropped. The only limited pool is for celebrity AI talent. | | |
| ▲ | lifeisloving an hour ago | parent [-] | | You should start a recruiting outfit then if its that easy, because you could be making 40-60k per hire!!! Your average Stats/machine learning expertise has little to do with standing up training infra and pipelines across $50 million in gpus. |
| |
| ▲ | verdverm 4 hours ago | parent | prev [-] | | you haven't cited any examples, nor acknowledged this is how everything in computing has gone, DK is a two way street I suppose stable diffusion is an excellent example, the price to train today's or yesterday's frontier is the analogy, many more people will be able to train up what we have today, especially as more of the pipeline is open sourced and the knowhow distills | | |
| ▲ | lifeisloving 2 hours ago | parent [-] | | >stable diffusion is an excellent example, the price to train today's or yesterday's frontier is the analogy, Dont use llms to draft your responses to people, extermely cringe. That explains the llm induced Dunning Krueger. What examples... There are no public numbers regarding how much it costs to train a 10T model, atleast nothing concrete. You should just use common sense and think about how you have to fit tensors in memory to train, and the larger the model is the more GPUs you need. Then you could also use common sense to think about how GPUs have gone up in price, not down. | | |
| ▲ | verdverm 42 minutes ago | parent [-] | | > Dont use llms to draft your responses to people, extermely cringe You are throwing around unfounded and incorrect accusations. Why? It's a weak ass rebuttal mate common sense is using things like history to guide expectations instead just saying things without evidence that are contrary to the patterns in computing history |
|
|
|
|
|
