I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt. Is this really going to stop anybody determined enough to make that kind of outcome?

There is an extremely narrow band of things that the AI shouldn't be answering, and that is generally immediately-actionable advice that allows someone to build something of harm to others. But even then, in an age where Tor, bittrent, i2p, abliterated local models, etc are freely available, let alone numerous books and online resources, is there even a point? Is it worth fully compromising the principles of free agency to an increasingly oppressed populace?

But instead of that we are handing the keys to regressive and repressive governments to order the suppression of any knowledge they deem inconvenient. I really doubt anyone is going to take a principled stance when the company's party minders threaten local staff with a rubber hose or incarceration.

I'm sure China et al are already doing this.

For the past 30-40 years humanity has received an incredible gift in these sand-powered thinking brainboxes. A gift that allows the common man to empower himself with a force multiplier towards his own success, and now access to superintelligence the likes of which few have ever seen. These can be tools to destroy the oppression that governs our lives from foolhardy, greedy, bootlicking control freaks. And here we are squandering it.

> These can be tools to destroy the oppression that governs our lives

So far it seems that the clearest use for these tools is to enhance, rather than destroy, oppression.

1. Suppression / elimination of white collar jobs

2. Negative cognitive effects, especially for young people

3. Accelerated decline in social media / information ecosystems. Increasing polarization, hard to tell fact from fiction.

4. Environmental impacts: increased energy usage means more carbon in the atmosphere, climate change accelerates.

5. Software security incidents increasing. Hard for individuals and small organizations to defend themselves.

6. “Power to think” vested in a very small group of organizations/labs. Doing work which should only require a computer and freely-available software will now be gated by expensive subscriptions. Once you “vibe code” a significant portion of your software you’re locked in and cannot go back to maintaining it without frontier-model level assistance.

Security theather is easy and gets lots of eyeballs. Actual security is hard and no one cares. Which one do you think soon-to-ipo companies are going to pick?

> I just love this whole "forbidden knowledge" schtick the AI safety dweebs have stuck up their butt.

It's just the latest incarnation of a timeless debate. In the 1970s and 1980s it was about the Anarchists's Cookbook, which was revived again in the 1990s when it started circulating on the Internet. There are many timeless debates, but the debate over weapon-making knowledge is much more concrete and predictable.