| ▲ | digitaltrees 6 hours ago |
| I actually think it’s the opposite. I think people will move back to their own hardware with OSS deployment like dokku. If everything is more bespoke the attack surface is smaller as is the pay off. Hacking railway got access to lots of api keys, hacking some random vps gets less payoff. |
|
| ▲ | roadside_picnic 3 hours ago | parent | next [-] |
| > If everything is more bespoke the attack surface is smaller This is exactly the problem that LLM assisted attacks solve. For a human attacker, the extra work to figure out a bespoke solution is too time consuming to be worth the effort. LLMs can search for vulnerabilities and spit out bespoke attacks like it's nothing. On top of this the bespoke solutions weakness leans heavily into the AI attackers advantage: response time. Even if they can just get your company's single Anthropic key, if they do it at 2am while your one infra person is on vacation, they have plenty of time to extract enough value for their effort. I agree with the move to hyperscalers. The future (well, actually present) is going to be constantly seeing new attacks happening. The only real solution to this going to be having teams of people and agents constantly watching for these and able to deploy fixes and respond to risks immediately. |
| |
| ▲ | digitaltrees 3 hours ago | parent | next [-] | | You’re technically right, which is the best type of right, but my point is the fragmentation and lower reward will make discovery harder, and the value of searching and cracking less worth the effort. I had a html game I built for my niece on railway with an open ai API key so she could vibe code changes. No hacker would have looked for it but now I am catching strays because railway was a juicy target. Hyper scalers would be the same. Some random colo not so much. | |
| ▲ | elevation 2 hours ago | parent | prev [-] | | > LLM assisted attacks solve Yes, LLMs can easily knock over your on-prem docker appliances. But you can still put stuff out of reach. I'm currently designing a key on-prem appliance that is can be compiled down to transistors, giving deterministic wire-speed performance while ruling out RCEs. It's cryptographically homomorphic: it processes authorized inputs but doesn't know what they mean, so an attacker who obtains the disk doesn't benefit either. The design will be highly resistant to exploits, and is likely to remain outside the "next-token" prediction training set. This is the kind of thing AWS won't sit down and engineer for you. But thanks to LLMs I finally have the time. |
|
|
| ▲ | itissid 6 hours ago | parent | prev | next [-] |
| How is that ever going to work out? The speed of discovery for LLMs is so fast, these places are getting breached more often than not due to a human errors. Its really hard for machines, let alone humans, to guard against a lot of LLMs going against them 24x7. |
| |
| ▲ | vb-8448 6 hours ago | parent | next [-] | | It's much easier to protect yourself if you have your infra behind a perimeter firewall you control. | | |
| ▲ | roadside_picnic 3 hours ago | parent | next [-] | | It's also harder to explain to your customers why you're the only product they use that has a security problem when an exploit does eventually happen. You see this already with uptime. If Github or AWS has a major outage, it's in the news and even people that don't understand software realize it's just "something happening today, everything is down". And, unfortunately, in practice customers don't care about this inverse, when "everyone else is down but not that one product". | | |
| ▲ | vb-8448 2 hours ago | parent | next [-] | | Good like trying to explain to your customers that their data is no more accessible because some api key ended up in a public git repo or since you admin panel is exposed to whole internet someone was able to log in. | |
| ▲ | frankchn 3 hours ago | parent | prev [-] | | Especially when the customers' own websites or services are down, they are a lot more understanding. |
| |
| ▲ | nradov 3 hours ago | parent | prev | next [-] | | Until the attacker bypasses your perimeter firewall by convincing an employee to download and install malware. | | |
| ▲ | mlhpdx 2 hours ago | parent | next [-] | | Having employees (or system like bastion hosts) with unrestricted access is one of the recurring mistakes in such events. I can’t understand it. | |
| ▲ | vb-8448 2 hours ago | parent | prev [-] | | And so? This is true for any kind of infra! You can't defend yourself from everything, but you can try to make you life easier by reducing the attack surface and make side movements more difficult. |
| |
| ▲ | 0xWTF 6 hours ago | parent | prev | next [-] | | Until there's a exploit for Palo Alto, Juniper, Cisco, F5, <pick your defender>. | | |
| ▲ | m4rtink 6 hours ago | parent | next [-] | | You can scratch Cisco from the list - they just announced they will re-vibecode all of their codebase! So there will not be any explots for their hardware in the future - one way or the other. ;-) | | | |
| ▲ | kube-system 3 hours ago | parent | prev | next [-] | | Or there's some exploit in an endpoint device, and the problem moves laterally from inside your firewall. | |
| ▲ | vb-8448 5 hours ago | parent | prev | next [-] | | Yep, but the attack surface is much smaller! | |
| ▲ | echelon 4 hours ago | parent | prev | next [-] | | My hate for hyperscalers is greater than my hate of being SRE. It's time to leave GCP, AWS, and the rest. Google and Amazon markup can eat it. Fuck those guys. Let's not forget that Google and Amazon tried to offshore us, played a role in inventing the labor-displacing models in the first place, and that don't give a shit about any of us. I hope they ossify and turn into the next IBMs. They thought AI would lead to growth and a greater concentration of wealth for themselves, but it's actually the seeds of their own destruction and dismantlement. | |
| ▲ | InsidiousWombat 5 hours ago | parent | prev [-] | | yeah remove the door locks, the thief can just break the window |
| |
| ▲ | buremba 5 hours ago | parent | prev [-] | | [dead] |
| |
| ▲ | ang_cire 4 hours ago | parent | prev | next [-] | | No self-service sign up for your app. No public access to the app itself. Website that let's you submit a sign up request form. Team reaches out and reviews your request, creates an account in the app dash for you, and you set up mTLS to get access to the dash. | | |
| ▲ | kube-system 3 hours ago | parent | next [-] | | LLM discovers exploit -- you don't realize it in time because you don't have a security team as big as the PaaS you used to use -- you get pwned anyway. This has happened to many high-profile self-hosted pieces of software just this past month. | | |
| ▲ | ang_cire 3 hours ago | parent [-] | | An exploit in what, the network stack of the Linux distro you're using for the mTLS proxy? Or the crypto module? Sure, could happen. But then, you're unlikely to be the top target of anything. Or do you mean a customer was running it? | | |
| ▲ | kube-system 3 hours ago | parent | next [-] | | There have been a lot of exploits for a lot of software on-prem tech stacks rely on just over the past month - proxmox, php, nodejs, palo alto, eMudhra, etc come to mind. > But then, you're unlikely to be the top target of anything. That would have protected you 3 years ago when a human had to bother to attack you. Today it doesn't protect you because it's trivial to deploy an LLM to act like an APT against any random target that shows up in shodan. It takes more effort to secure on-prem software today than it did just a couple of years ago. Updates are more frequent, zero days are more frequent, attacks are more common, more persistent, and more automated. I think most people responsible for securing production systems today will share the experience that their workload has increased. | | |
| ▲ | ang_cire 3 hours ago | parent [-] | | You're still describing an environment that presents a pre-auth surface area beyond a listening port. And I am literally a security engineer; the work has absolutely not gotten harder, it's just changing in how you think about security, just like it shifted before from "secure the perimeter" to "defense-in-depth", it's now changing again to focus on automated response and compartmentalization/ segmentation (and I mean real segmentation, not the checklist-style "we have 3 vlans, but they can all talk to each other" 'segmentation' you used to see. | | |
| ▲ | kube-system 3 hours ago | parent [-] | | Yeah, most organizations need to expose something to the internet as a matter of normal operations -- at least some sort of a web app if not dozens. It would be great if we could all just have udp 56820 listening silently for a key, but that isn't reasonable for most organizations that have normal users expecting normal applications. Yes, everyone is rushing to implement zero-trust principles for real now. It's getting harder to trust that your public facing endpoints don't have vulnerabilities, and it's getting harder to trust that your internal software and endpoints haven't been compromised from within your pre-existing trust boundaries. A lot of the increased workload is due to patching up the existing security infrastructure, and on top of that, building out new solutions to better fit today's threat landscape. | | |
| ▲ | ang_cire 3 hours ago | parent [-] | | Ideally they need to divide those apps entirely; your public website doesn't need to talk to your app. For stuff like Netflix that is truly consumer-facing, where mTLS or IP whitelisting are non-starters, the reality is you have to rethink your model entirely. Also, I'm not sure where you got the idea I'm talking about on-prem only; this also applies to using IaaS/ cloud hosted infra. | | |
| ▲ | kube-system 2 hours ago | parent [-] | | The reason we were talking about on-prem is because we're talking about the effort of doing these things yourself vs farming it out to a third party. If you're a security engineer responsible for securing only: 1. a public website 2. stuff that can be ip whitelisted and/or use mTLS Consider yourself very lucky. Most orgs need to expose a lot more than that. | | |
| ▲ | ang_cire 2 hours ago | parent [-] | | Almost no companies farm their security out to third parties. The era of MSSPs is long dead, except for small non-IT businesses like mom-and-pop law firms. > Most orgs need to expose a lot more than that. No, in fact there are only 2 kinds of connections: 1) public, necessarily allows anonymous/unauthenticated connections 2) non-public, doesn't need to allow unauthenticated connections Name a service, and I can tell you which one it is. Or put another way, there are authenticated and unauthenticated services. If you can authenticate someone at the application level, you could do it at the network level. Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it. And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting. SFDC? mTLS. Customer webhooks? IP whitelisting. Data warehouse? PtP VPN. | | |
| ▲ | kube-system an hour ago | parent [-] | | > Almost no companies farm their security out to third parties. They do it all the time, piecemeal, that's part of what anything-as-a-service delivers. If you don't want to secure an app, buy a SaaS one. Don't want to secure infrastructure?, buy an IaaS. > Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it. > And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting. Literally any SaaS software where the customer does not or cannot practically install a certificate for you or provide a list of IPs in advance? mTLS is almost never used in a browser unless you are connecting to a government website with a pre-shared smartcard. IP whitelisting is great if the end users only work from behind a VPN or in an office. But there's a whole big world of people who cannot or will not do that. | | |
| ▲ | ang_cire an hour ago | parent [-] | | And that is going to have to change. There are already plenty of methods that are low-impact to users, like certificate-pinning in apps, and all you have to do to make it per-user is roll in their user-specific key JIT when they download the app. Users will not be happy (just as no one is happy Reddit is going app-only), but they will do it, especially when it becomes common as companies decide they don't want to be the 5th breach notification on HN in a given day. Today we have Bun, Cursor, Northflank, and now Linode! |
|
|
|
|
|
|
| |
| ▲ | sdenton4 3 hours ago | parent | prev [-] | | Or the configuration parameter that you got wrong because you rolled your own and didn't know any better. | | |
| ▲ | ang_cire 3 hours ago | parent [-] | | Why would you roll your own? | | |
| ▲ | kube-system 3 hours ago | parent [-] | | You would inherently be rolling your own configuration when deploying something on-prem. | | |
| ▲ | ang_cire 2 hours ago | parent [-] | | 1) where did you get "on-prem" from? 2) that's not what "rolling your own" means. Building your own web proxy is "rolling your own". Standing up e.g. Caddy and configuring it is not. | | |
| ▲ | kube-system 2 hours ago | parent [-] | | 1. that's what this thread is about?: > I think people will move back to their own hardware > self-hosted pieces of software 2. arguably -- but lets set aside what you think the word means and respond to the obvious meaning of the comment you're replying to. The point is -- doing shit yourself makes it your problem to do correctly and requires sufficient resource allocation | | |
| ▲ | ang_cire 2 hours ago | parent [-] | | 1. That's fair, I didn't realize people were expecting my comment to be perfectly-aligned to every aspect of the thread parent, including the "their own hardware" part, but now I at least see where the confusion came from. For the record, any issues at this level of the stack are also present in IaaS, not just on-prem. 2. I still don't see where you got that they are only talking about configurations versus messing up an implementation (I.e. in a paramater they coded incorrectly)... But that aside, even on AWS you're still configuring your proxies, gateways, load balancers, etc. AWS gives you just as much rope to hang yourself with as a box in a colo network cage does. Companies are already doing this stuff themselves, on-prem is not more security work. IaaS is about scaling and maintenance load, not security being handled for you. |
|
|
|
|
|
|
| |
| ▲ | clickety_clack 4 hours ago | parent | prev [-] | | In-person requests for access to any app. Request interview includes word association exercise that monitors your emotional reaction to a string of key words. | | |
| ▲ | itsdesmond 4 hours ago | parent | next [-] | | Security question 2: You reach down and flip the tortoise onto its back… | |
| ▲ | ang_cire 3 hours ago | parent | prev [-] | | You joke, but the space I work in actually does do video calls (or in our case in-person meetings if they're local to SF, often) before onboarding new customers. |
|
| |
| ▲ | ijustlovemath 6 hours ago | parent | prev | next [-] | | For one, I have strict iptable rules to drop inbound from outside my specific commercial subnet and my specific residential IP. After that, all the usual best practices. | |
| ▲ | nine_k 4 hours ago | parent | prev [-] | | Breaking a lock still costs approximately the same, whether the lock is protecting a small barn or a huge hangar. A large number of small but reasonably tough (not unbreakable) targets makes the payoff per target smaller. This necessitates target selection, and chances of being deliberately skipped grow for an average target. | | |
| ▲ | TeMPOraL 3 hours ago | parent | next [-] | | However, the large hangar is likely to have armed guards inside, or at least able to arrive on the scene within two minutes of the alarm tripping. Your regular household with their regular locks on their regular doors does not. | |
| ▲ | digitaltrees 3 hours ago | parent | prev [-] | | Everyone knows Fort Knox has gold, and where it is, that’s railway, heroku. No one knows where my personal safe is or whether it has anything worth taking. That’s a vps in a colo |
|
|
|
| ▲ | kube-system 3 hours ago | parent | prev | next [-] |
| This is misguided thinking. You are less likely to hear about on-prem breaches in the news just because they are less likely to be newsworthy incidents, but they are even more likely to happen. Part of the reason people moved to the cloud is because it is much easier to have a team manage updates and security at scale. Being a small target is not helpful these days, attacks on on-prem software today simply scans the internet, or uses databases by people who have already scanned it, and is automated. On-prem software has not been spared from the recent spat of security incidents. |
| |
| ▲ | heisenbit 2 hours ago | parent [-] | | There is a difference of the security of software running in your cloud resources rented from AWS etc. managed by you and being a tenant in many SaaS systems all cobbled together all running on rented hardware. There is a lot more attack surface and a lot more diffusion of responsibility. |
|
|
| ▲ | burnte 4 hours ago | parent | prev | next [-] |
| This is absolutely happening. I'm not a developer in SV, I'm an IT leader out on customer land, and I can tell you a LOT of companies are looking to bring things back on prem for security and cost reasons. And before anyone laughs, yes, you ABSOLUTELY can save money going on prem, I've done it for company after company. It's one reason the PE firm that I've worked under for a decade kept moving me to new companies. |
| |
| ▲ | digitaltrees 3 hours ago | parent | next [-] | | Let them laugh. Self hosting is much easier with OSS that solves most of the pain that made the cloud transition worth it, and commodity hardware has been cheap and capable for over a decade. | |
| ▲ | TedDoesntTalk 4 hours ago | parent | prev [-] | | My company just did this. Still using AWS S3 but everything else is on-prem with kubernetes. | | |
| ▲ | eddythompson80 3 hours ago | parent | next [-] | | That sounds insanely expensive. AWS S3 only makes sense if your workload is also in AWS. Using AWS S3 while having your compute elsewhere is a recipe for an insane egress bill. | |
| ▲ | yurishimo 4 hours ago | parent | prev [-] | | How much of your public facing workloads rely on S3 if I can ask? | | |
|
|
|
| ▲ | Imustaskforhelp 6 hours ago | parent | prev | next [-] |
| My comment[0] tries to explain the nuance and I would agree with you and both the GP in some sense. For Entreprise, the shift will be towards hyperscalers as they want trust and less blame (Nobody got fired for picking AWS) For consumers, we care about pricing and stability and control and we will move back to our own hardware or cheap VPS providers and self-host things. The middle ground would be where the slight pain could be at in my opinion. [0]: https://news.ycombinator.com/item?id=50307410 |
| |
| ▲ | happyopossum 5 hours ago | parent | next [-] | | > For consumers <snip> self-host things. That's not how consumers work. That's a nerd/geek/hobbyist mentality. If Ikea and Wayfair went out of business you wouldn't expect consumers too take up woodworking and build their own furniture en masse. | |
| ▲ | SoftTalker 6 hours ago | parent | prev [-] | | For consumers/small orgs, how is the security at a cheap VPS provider going to be any better than at a cheap IaaS provider? | | |
| ▲ | collabs 6 hours ago | parent [-] | | as long as you follow the herd (for example, disable password ssh login, only allow ssh key login, use something mind numbingly boring like debian stable, set up unattended updates, update and reboot every night, etc, don't use any exotic packages etc) I suspect you are simply not a worthwhile target even with LLM agents as opposed to humans | | |
| ▲ | SoftTalker 5 hours ago | parent [-] | | Until your VPS provider has its management dashboard breached, allowing an attacker remote console and configuration access to your VMs and everything that enables. | | |
| ▲ | jodrellblank 4 hours ago | parent [-] | | I dunno why you're getting downvoted, they don't even need their control plane breached, it could be a neighbouring VM running something out of date and an attacker getting out to the hypervisor and taking your disk images without you or the hosting company ever knowing. | | |
| ▲ | digitaltrees 2 hours ago | parent [-] | | Then buy hardware and go colo. a dell server off eBay would far exceed what most smbs need. |
|
|
|
|
|
|
| ▲ | debo_ 3 hours ago | parent | prev | next [-] |
| It's probably both. |
|
| ▲ | wat10000 4 hours ago | parent | prev | next [-] |
| Cost of hacking a custom setup is going down rapidly. There was broad agreement that the PHP vulnerability was very difficult to exploit in practice. It took two hours for someone to get into Steal My Bitcoin. Apparently all they did was tell a decent model "go exploit that." AI isn't free but it's not that expensive. |
| |
| ▲ | TedDoesntTalk 4 hours ago | parent [-] | | If you have to hack 1,000,000 custom setups to get the same results as one hyperscaler, would you bother? | | |
| ▲ | wat10000 4 hours ago | parent [-] | | Depends on what takes more effort. It's not at all clear to me that the 1,000,000 custom setups takes more effort. The first few will take some, then you'll start to see patterns and it'll get easier as you go. Custom setups aren't going to be all that unique in practice. |
|
|
|
| ▲ | malux85 5 hours ago | parent | prev [-] |
| No thats an old way of thinking because it assumes mass targets are better because hacking is a high effort activity. But with the automation of intellect the unpatched systems (which a much more numerous when everyone self hosts) is the key point. Mass credential stealing is just the top funnel of hacking and (compatativly) isnt that lucrative - these credentials are then sold on the black markets to the next step of hackers who use them to exploit targets. With unpatched systems and self hosting, this second group can just set an llm at a target and have a much higher chance of success without needing credentials It would be a dream for the infiltration hackers to have everyone self host. Unpatched and miscondigured systems would be everywhere and LLM automation could make mass scanning them and exploiting them very attractive - not mass scanning and exploiting them for credentials stealing, no! For layer 2 stuff |
