What he really means is:

Please regulate us, now! If we don’t get regulatory capture ASAP I’m toast. I mean look man I just had to go beg for tens of billions more cash so I can keep the lights on and my IPO is looking more and more screwed every day. Please, I’m begging you. What more do I need to say to convince you?

"To stop our all powerful AI that will go rogue you need to buy our all powerful AI"

This is the textbook definition of racketeering.

Not even the singularity can fix php.

But all jokes aside, we're entering a new era where there's a paradox of pro-AI people going AI is really good, but not that good... when it kind of is that good.

We'll probably have ai monitors as a service, but the bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.

> OpenAI is apparently getting into the arms dealer model of selling to both sides of a conflict.

Business is booming!

"Altman seems to be scolding these organizations for not taking the situation seriously enough or considering that models might evolve and improve, but seems to have less interest in reflecting on whether or not he might have undersold the issue while pushing forward"

Meanwhile, look back at the old Mythos and Glasswing threads and all of the AI haters on Hacker News were lambasting Anthropic for hyping up the next token generators that could never find a bug not in their training data...

The eternal cycle of discovering that engineering is hard continues.

"As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs."

Maurice Wilkes 1949

Patch all the bugs? Does that mean you can find all the faults in a system? I don't think so. At best, you keep up with known bugs and exploits.

My understanding is that Kurt Gödel told us you can't make any system completely safe from intentional misuse. So, yes we can patch known exploits, but there will always be unknown exploits.

Perhaps AI's can help monitor for new exploits as soon as they are attempted?

Please feel free to explain further, or even tell me I'm all wet. I guess this is one of those times I'd really prefer to be wrong.

OK, SamA, if things are that bad I have a proposal. Listen carefully, your business and future depend on it.

Meet the new net, superficially the same as the old net

How to tackle the bug backlog? It is fairly simple, we wind back the clock on network services all the way to the dawn of the 'net, replacing known-insecure things like telnet, ftp and finger with heavily audited alternatives based around ssh etc. The rule is that only audited software is allowed access to the 'new net'. Of course none of those newfangled agents and models are audited not can they be audited in any real sense of the word so unfortunately for SamA they're banished to the 'old net'. The idea would be to get the 'new net' up and running quickly and move critical services which need to be network-reachable there. It needs to be bilt in such a way that even when one of SamA's toys gets access to the 'new net' it can't do much, or anything. The 'new net' could be compared to a 'mission critical API' to essential services, useable by all who need to have access but technically off-limits to SamA's creations. They are available on the 'old net' where all bets are off and which no longer can be used to interface with critical services 'cause they're not connected and can not be connected.

Now all that's needed is to find a way how to implement this without bugs, without handing the ultimate control mechanism to power brokers, without loosing too much real functionality, without ... well, many problems. When the enemy is inside the gates half-measures tend not to work, best find him and lop his head off. Better be careful where you stick your neck, SamA.

"Too dangerous not to regulate!" Nobody should have general compute!

I hope Sam altman reads HN comment sections

Nothing we're seeing so far compares with the summer of 2003. It may soon! It may be much worse! But you can tell that a lot of people in the discourse did not practice in this field 20 years ago. We should be clear-eyed, but not yet be destabilized by what's happening.

AI is, apparently, very good at producing proofs. Formal verification of program properties was out of the reach of human labor, but may not be out of reach for 10 million instances of Claude. Software security will, by my estimate, just shift to this new paradigm.

The more likely thing is that we'll see people use the sandboxing and security tools that people already have. After all, it's quite easy to get LLMs to write a seccomp profile, a Kubernetes manifest, or a systemd unit with the minimum required privileges and permissions, and for insecure C code, we'd probably see mainstream use of Fil-C and similar.

Unrealistically, sometimes I wish we’d have a “year of bugfixing” - where no new features are allowed across the entire industry. Heck, even a month of bugfixes could do a lot of good.

When is the blackwall going live?

I'm going to predict right now, as I weep into my coffee, massive centralized AI traffic monitoring is the only way that we fix this. The attack surface is effectively infinite as you look across all of the home routers, Keurigs, iot's, and mobile devices - and security has historically ranged from reactive to uninterested by people, companies, and government. I see no world in which my parents proactively do anything other than unplug the router to help protect against the coming wave of exploits. So the only way I see us getting a handle on protecting that infinite uneducated and soon-to-be helpless landscape is through centralized inspection and policing nodes. And dear God, what a dystopian future that looks like to me (˃̣̣̥ᯅ˂̣̣̥)

what a dream. sam is selling the cure (tokens) and disease (new model releases). he just cranks up that disease when he needs to sell more of the cure. ceo of the decade.

With php2shell going mainstream, to the point that random friends who happen to know I'm a programmer, consult me to understand the extent of it, I think they succeeded with their doomerism message.

When we find a bug, we patch it. When we find more bugs, we patch them as well. That's what happened here. Sam is telling us, "we found bugs, it's over" and the general public swallowed the frog.

I mean, there is the bbc interview where they invited a cyber security expert who said he didn't know php existed before the the news broke out. The general public doesn't see that gaff.

I think I can guess what his proposed solution will be.

Maybe instead of patching all the bugs we'll instead leverage LLMs rewrite all the software from scratch with fewer exploitable vulnerabilities. That would have been a stupid idea a year ago but now it seems increasingly possible.

Obligatory “We are all trying to find the one who did this!”

Sam Altman is definitely not Oppenheimer.

He is correct that everyone needs the models not just the offense that hacks their way into Mythos while the rest of us are left locked out and hoping for the best. But that statement is ALSO self-serving.

Kinda wish he’d gone into investment banking.

We spent decades creating those bugs. Give it few years at least to find them and patch them.

There likely will be no way to defend the increased rate of cybersecurity issues due to AI, without incorporating AI in the response.

The new patching and security fixes might be perpetual hotfixes.

This could mean tough times for the cloud in some ways.

Regulate the bugs

Yeah, the staggering amount of CVEs in qmail and postfix can never be remedied. What a clown. Has he ever programmed seriously?

The strongest potions, you cant handle them.

Whatever one may think of AI, it's certainly showing the sh*ty shape normal cyber security is in.

I mean, the various supply-chain exploits we've seen said "we allow a dubious third party to blindly update our server 'cause it saves us admin money" and that speaks to how much other shoddiness a significant portion of companies allow on their servers.

It would be better if grant distribution relied on something saner than the effort to write a grant application, if education didn't rely the effort to write poor quality essays and servers didn't rely on the effort required to find the bugs you know are there. But those ships have sailed and I guess we're all just at sea.

If they actually think that this version of AI is that dangerous, then the real risk is that USA and China will work together on a joint "AI non-proliferation" agreement much the same way that nuclear weapons technology became restricted by those that already had it. This will lead to a de facto techno-colonization of the rest of the world just between themselves.

We already saw Trump and Xi discussing this exact scenario last month during their summit.