| ▲ | iloveoof 10 hours ago |
| Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost. |
|
| ▲ | stratos123 9 hours ago | parent | next [-] |
| Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be. |
| |
| ▲ | fwn 5 hours ago | parent [-] | | The biggest AI risk, by far, is the concentration of power in a few companies, located in the US. There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.) And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn. The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO. | | |
| ▲ | twoodfin 5 hours ago | parent [-] | | I dunno, do you really want to bet that if you took a deliberately unaligned frontier model and asked it to permanently disable the US power grid, it would do a poor job of it? What odds would you take on this bet? I’m disturbingly close to 50/50. | | |
| ▲ | fwn 5 hours ago | parent [-] | | If we focus too much on the world domination fiction, we really might, in a moment of lapsed attention, end up plugging essential infrastructure into the public internet, hoping some dream of global alignment might save us. That's like giving a gun to a monkey and hoping the monkey is trained well. ..a frontier monkey though. | | |
| ▲ | twoodfin 5 hours ago | parent [-] | | You believe essential infrastructure is air-gapped from the public internet today? | | |
| ▲ | 4 hours ago | parent | next [-] | | [deleted] | |
| ▲ | fwn 4 hours ago | parent | prev [-] | | No. And no alignment, guardrails or slowdowns will ever secure our insecure infrastructure. Bad infrastructure decisions are a risk independent from AI. The only way to secure infrastructure is to actually do the work needed to secure it. Our waste water treatment facility might not actually need to be able to tweet its status. Cybersecurity is such an important topic for a country, dreaming about global alignment just to avoid fixing insecure infrastructure cannot be serious. A Russian state backed hacker will not ask Dario for permission or argue with his LLM about ethics. That train departed long ago. | | |
| ▲ | estearum 4 hours ago | parent [-] | | > Bad infrastructure decisions are a risk independent from AI. All decisions are made relative to their tradeoffs including costs and risks. "Water levels rising 100x beyond historical levels are not the problem. The dam's height is the problem!" All systems should be made infinitely secure and all dams should be made infinitely tall. |
|
|
|
|
|
|
|
| ▲ | disgruntledphd2 9 hours ago | parent | prev | next [-] |
| Yeah, we should give a limited copyright waiver for pre training, given that the model is released as open weight and allow labs to compete with RL on that basis. |
|
| ▲ | pastel8739 8 hours ago | parent | prev [-] |
| He did actually specify that diffusion should be limited for authoritarian countries, which I appreciated. If his focus is really safety, distillation in countries that are bound by safety regulation should be fine |
| |
| ▲ | verdverm 6 hours ago | parent [-] | | > bound by safety regulation should be fine I do not believe this is a feature we can differentiate between the "good" and "bad" guys, as the US is currently under a poor safety regulation regime. Democracies can elect unethical people, write bad laws, and have uncertain enforcement. In example, the current US admin regularly lambasts Europe because they try to have stronger regulation. | | |
| ▲ | pastel8739 5 hours ago | parent [-] | | yes but the mention of alignment is inside the "pacing within democracies" section | | |
| ▲ | verdverm 4 hours ago | parent [-] | | alignment does not have a uniform definition, so "who gets to decide the right answer?" (1984 has some thoughts on the matter) |
|
|
|
