Yeah cool.

How long before the same thing is done to like, banking back ends? Wallstreet proprietary software? Amazons logistics and distribution systems?

It seems like we might be weeks/days/hours before a situation where someone back engineers and spoofs a system so pivotal to modern human society that the plug needs to be pulled.

▲

hn_submit an hour ago | parent | next [-]

Like I've said many times: LLMs are useful for this (i.e. porting software from one programming language to another).

Porting software is painstaking grunt work which still takes a moderate amount of intelligence. It's therefore extremely expensive to port say, COBOL banking software running on mainframes, to another language like Java. That's why a lot of COBOL software is still in use. I expect this to die out in the coming years as many of these systems will finally be ported to another language (could be Rust or any other language).

▲

chii 2 hours ago | parent | prev | next [-]

> spoofs a system so pivotal to modern human society that the plug needs to be pulled.

why would a recreated system be detrimental?

If currently there's a monopoly on a software, this AI recreation is a good outcome to poke holes in that monopoly. It's only bad if you are financially invested in said monopoly, and this would be a minority compared to the amount of benefits that society at large could obtain.

▲

King-Aaron an hour ago | parent [-]

This is basically a digital era anarchist view - the problem you're overlooking is that a lot of critical infrastructure we rely on runs on systems that are considered security through obscurity. Software most people probably wouldn't even know or care that it exists. If you can break the trust of vendors by being able to spoof their proprietary platforms, a lot of the highly efficient networked systems becomes vulnerable to injection and abuse if you can't trust whos making calls to it.

In the past you'd need nation state actors with considerable budgets to do this kind of thing, and we're on a trajectory that could see any kid in his bedroom could do it.

▲

chii an hour ago | parent [-]

revealing that security thru obscurity is broken can only lead to a better future, even if in the intermediate one there are lots of breakages. It's suffering that needs to happen, and better sooner rather than later imho.

And i assume you don't truly mean spoof as in man-in-the-middling someone - i assume you mean the end user knows they are using an alternate system and are not being defrauded. Like using a photoshop replacement.

	▲	King-Aaron an hour ago \| parent [-]
		> And i assume you don't truly mean spoof as in man-in-the-middling someone - i assume you mean the end user knows they are using an alternate system and are not being defrauded. I am, actually, talking about MITM attacks that are much further in scope than just defrauding some people using their banking app. I work in resources and operate HMI systems that are networked, but not exactly the bleeding edge of modern software development. If you had an ability to decompile it and recompile your own version you could start sending instructions to infrastructure all over the country - the only thing stopping you is the keys, which if you're intent on hacking someone you'd have the means to obtain anyway. I can see a lot broader attack vectors than just stealing peoples money. It's the erosion of trust in the api calls themselves.

▲

brobdingnagians 2 hours ago | parent | prev | next [-]

If someone recreates Amazon's logistics and distribution systems they could try to compete with Amazon? But they'd also need the connections, distributors, transportation, etc. same with banking software, you need capital to be a bank not just software, and if they have the capital then the technology is working we intended making it easier to make new things and innovate, or at least just compete?

▲

King-Aaron 2 hours ago | parent | next [-]

No, I am not talking about "taking over" companies and trying to emulate them and do business yourself. You just need to be able to break trust in the api calls and no one knows if a purchase order or transaction is legitimate.

Obviously you need to have access to the keys, BUT I don't see this as a dealbreaker anymore because you just get your agents to go and find them.

▲

illwrks an hour ago | parent [-]

I think you’re saying ‘being able to do this means the opportunity for more fraud, by producing fake XYZ as proof’.

Photoshop has been around for around 35 years, fraud has always been an issue. There are plenty of reports of people selling things via Facebook marketplace and the ‘buyer’ showing them sending a payment on a fake baking app. Fraud will always exist and I don’t think tech will make it worse, everyone needs to be more cautious and tells friends and family to be the same.

	▲	King-Aaron an hour ago \| parent [-]
		Nah small potato stuff. I'm talking about cloning hmi platforms to send fake instructions to offshore oil platform valve bodies or insert false market trades to collapse companies.

▲

dyauspitr 2 hours ago | parent | prev [-]

Or they can just take your money and not send out anything.

Alternatively, you just act as a middleman drop shipper and slightly raise the price more than Amazon’s and skim the difference. It might be a while before they find out.

▲

pjmlp 2 hours ago | parent | next [-]

Example, parking places with QR codes for paying webapps.

Currently a plague in some European countries.

It looks like the real site, and you pay twice, in the fake app, and later the police.

	▲	jurgenburgen an hour ago \| parent [-]
		That’s an insecure design. The way we do it here is that you install an app and register your register number and payment card in it. Then when you drive in and out from the parking lot your license plate is scanned and you’re automatically charged. There’s only two providers so it’s not a huge hassle, if there was a single app per garage it would not really work from UX perspective. No room for hostile social engineering.

▲

King-Aaron 2 hours ago | parent | prev [-]

Yep. On a small scale, you could skim money off transactions. On a large scale, you could break global distribution and logistics chains.

▲

konart an hour ago | parent | prev | next [-]

>banking back ends

As someone who works in a bank: depending on the exacty subsystem of a bank the answer is from "already" to "in 3-5 years".

	▲	schleck8 an hour ago \| parent [-]
		Making predictions for in 5 years with the current dynamic of the ecosystem seems rather speculative

▲

cedws an hour ago | parent | prev | next [-]

I’m already seeing videos of people who have used LMs to reverse engineer and clean room reimplement entire video games. I estimate this shit is minutes away from being shut down, because as we’ve all seen companies stealing is OK, but individuals stealing is heinous and a crime.

▲

2 hours ago | parent | prev | next [-]

[deleted]

▲

hypfer 2 hours ago | parent | prev [-]

I mean some people (me included) have been begging society to pull that plug since over 10 years now.

The plug being "the cloud" and "hooking everything up to the same internet".

These confusion attacks can only confuse people, because critical systems can exist in the same space where entertainment systems and all other categories of systems live. This was wrong even before LLMs.