| ▲ | pudgywalsh 8 hours ago |
| Sadly this instantly became a political football, with the states pointing fingers at Iran, but Trump was not wrong in this case. This is gross incompetence at all levels — IT malpractice if you will. CISA and its predecessors have been warning utility operators about critical infrastructure vulnerabilities for what, 15 years at this point? That goes back to the first Obama administration. Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords. You cross a threshold where you're being deliberately careless. When you are putting more effort into securing your Plex server on your home network then public utilities are taking on machinery that dumps chemicals into the local water, something is not right and finger-pointing isn't going to fix it. |
|
| ▲ | Avicebron 8 hours ago | parent | next [-] |
| I think it's less carelessness and more the inability to attract (pay) people who have the technical knowhow to properly secure infrastructure. Even a lot of developers are poor network engineers and treat IT like magic at their own companies. |
| |
| ▲ | pudgywalsh 8 hours ago | parent | next [-] | | I've met info-sec / vulnerability researcher types that were egregiously reckless, like plugging Raspberry Pi's into the production network kind of thing. Public sector has always paid low. But the problem is widespread, almost universal, and they've had a 15 year head start of the federal government telling them to get their shit together. At some point it just became standard industry practice is my guess. | | |
| ▲ | doobiedowner 7 hours ago | parent | next [-] | | Upgrades to waste water are project based. Lowest bidder will not provide security for free. Security may be mentioned in spec but in hand waved language that can be hand waved away. That company doing the improvement project will have next to no documentation from the previous engineering effort. Just do bare minimum and move on to next job, because no one is getting paid enough to do put in more effort. | |
| ▲ | elictronic 6 hours ago | parent | prev [-] | | In all things when it fails in a drastic way the system will be corrected. People will die, it will suck, changes will be made. Government is supposed to respond to these things and create incentives to correct. Telling a small municipality to do something without a carrot ir stick does nothing. In this instance someone else will be providing the stick. |
| |
| ▲ | moscoe 8 hours ago | parent | prev | next [-] | | You’re on the right track, I think. But, I wouldn’t say it’s about the pay to attract competent workers. I think it has more to do with the incentive structures once you’re in. Incentives and performance management are fundamental problem in civil service. The incentives to set high standards and hold individuals accountable simply do not exist. The one and only exception is the military, because lives are literally on the line in a way that is not at all abstract. | | |
| ▲ | Avicebron 8 hours ago | parent | next [-] | | Usually when people say this they are dog whistling privatization. Which is the exact opposite thing people need in infrastructure, ask anyone who has to deal with PG&E. Paying skilled people highly does actually incentivize people to do better work, especially if they are actually embedded into the community they are essentially working for. If being a civil servant was as "glorious" as being a techie is SF there would be a very different attitude around the work. | | |
| ▲ | moscoe 7 hours ago | parent | next [-] | | In an environment without accountability, higher pay is just more incentive to lay low and not take any personal risk. Why do anything other than the bare minimum when there is no upside? In a bureaucracy where responsibility is diffused and the culture is purely political and not merit/performance-based, few are willing to step out of line to do the right thing. The people who climb to the top aren’t the ones who took a risk and got the reward. Quite the opposite, they’re the ones who learned to play the game and didn’t upset the power structure by rocking the boat. No one is going to tell the emperor he has no clothes when the path to power is political and has no grounding in reality. Highly conscientious, intrinsically motivated people will do the right thing in any environment. And those people don’t last long in political/bureaucratic environments where the incentives are misaligned. | | |
| ▲ | doodlebugging 6 hours ago | parent | next [-] | | Your comments show, beyond a shadow of a doubt, that you have never worked for the federal government and likely have never worked in state or local government. >In a bureaucracy where responsibility is diffused and the culture is purely political and not merit/performance-based, few are willing to step out of line to do the right thing. Federal employment is merit based. Advances are earned, not doled out to the teacher's pet or the ass-kisser with the most perfect pucker. You have no idea what you're talking about. I have no idea where you or anyone else got the idea that a government employee has no responsibility or incentives and works strictly for political ends and even worse, where none are willing to speak up. You clearly have no experience in that space and should focus your comments on things that you better understand through direct experience. >And those people don’t last long in political/bureaucratic environments where the incentives are misaligned. This is bullshit and you are grossly misinformed. Plenty of people fight the polarization of the system on political lines from within the system. They use the laws that are in place in order to prevent systemic abuses. They may not win every internal fight but they do go to battle for the right reasons. You just have no idea. | | |
| ▲ | fc417fc802 6 hours ago | parent [-] | | I don't think your attitude here is in keeping with the guidelines (or constructive discourse for that matter). You've made a number of uncharitable assumptions about the other party on the back of which you then launched into baseless personal attacks. Notably everything in the comment you replied to applies equally to the public and private sector. They are neutral observations about systemic motives and the associated perverse incentives. Also it's not clear to me that any of the utilities in question have anything to do with the federal government so I'm not sure why you dragged them into this. | | |
| ▲ | doodlebugging 5 hours ago | parent [-] | | >I don't think your attitude here is in keeping with the guidelines (or constructive discourse for that matter). I'll make a note of that. Any comment that I make on this site is made in the hopes that there will be discussion generated. Sometimes that happens and other times it doesn't. I don't spend any part of my life wondering why people choose to or choose not to engage. Most of us have busy lives and this (HN) for us is an opportunity to catch up on interesting things. >You've made a number of uncharitable assumptions about the other party on the back of which you then launched into baseless personal attacks. I am not sure that noting that a poster has no experience in a subject and has no idea how it really works in practice becomes an uncharitable assumption. It may look like a personal attack but it is based entirely on the content of the user's own posts, which support the conclusion that I reached. >They are neutral observations about systemic motives and the associated perverse incentives. From /u/moscoe original comment in the thread: >Incentives and performance management are fundamental problem in civil service. The incentives to set high standards and hold individuals accountable simply do not exist. These two sentences here demonstrate that /u/moscoe does not have any experience working in the public sector for federal, state, or local government entities yet somehow they feel qualified to speak confidently (incorrectly) that those who have worked in that capacity are ham-strung by the system and disincentivized to make waves. That is false. I mentioned federal employees specifically because they are civil servants (referenced by /u/moscoe above) and someone close to me has worked a long career in the federal government under several administrations from both political parties. I think you have your opinions and you are entitled to those opinions. I am also sure that you are wrong about the content of my post. It was not uncharitable nor were there any assumptions. The conclusions that I drew were supported by the statements made by /u/moscoe. If it ends up looking like a personal attack then there was a basis for that personal attack. It's possible that you read my comment without reading the full thread so some of the context was missing. |
|
| |
| ▲ | throwaway173738 4 hours ago | parent | prev [-] | | Corporations have even less accountability to the public than public utility districts. If you don’t like what the organization is doing, vote in different commissioners and make them aware of your concerns. But if you don’t like what your water service is doing, good luck drilling a well to get away from them. |
| |
| ▲ | fc417fc802 6 hours ago | parent | prev | next [-] | | Why do you assume that PG&E has a good incentive structure, or that the public sector must necessarily have a bad one? My only objection to the preceding comment would be that struggling with perverse incentives isn't limited to civil service. Incentive structures are a core struggle of approximately all large groups of people. | |
| ▲ | mlyle 5 hours ago | parent | prev [-] | | I think the big problem is administrative capacity. There are better run governments than we have in the US. The contempt for the state is a self-fulfilling prophecy. The state is incompetent because many of us believe it is inevitable that it will be. Compensation is just a part of it; coherent administration with continuity is even more important. |
| |
| ▲ | cyanydeez 7 hours ago | parent | prev [-] | | We just started replacing our PLCs. They absolutely were setup with default passwords, but weren't put on the public internet. | | |
| ▲ | fathermarz 6 hours ago | parent [-] | | What policies do y’all have in place for this? Is there a program in place or the beginnings of one at least? |
|
| |
| ▲ | arionhardison 6 hours ago | parent | prev [-] | | [flagged] |
|
|
| ▲ | andyjohnson0 8 hours ago | parent | prev | next [-] |
| > Yet here we are in 2026 and these utilities are still connecting these things to the raw Internet with default passwords. I work with PLCs. Default passwords of not, the idea that such weakly secure devices are being made accessible from the public internet boggles my mind. |
| |
|
| ▲ | fathermarz 6 hours ago | parent | prev | next [-] |
| Not IT malpractice and this where the industry diverges. IT folks usually don’t work on or understand these systems. Which is one of MANY problems OT faces. IT best practices don’t suffice in OT and even when they do, most of these orgs are too resource hamstrung to do anything about all of the fires they have to put out. Not to mention all of the OT vendors who flooded the market with tools instead of people being taught the boring process driven work. |
|
| ▲ | lorreyfum 7 hours ago | parent | prev | next [-] |
| Absolutely 100% spot on. It’s not a political issue, it’s a technical issue. Disconnect them from the internet. Run your security patches. Check your logs. Water supplies are pretty important, do your job. |
| |
| ▲ | fathermarz 5 hours ago | parent | next [-] | | “Run your security patches” is easier said than done in the case of OT and it’s actually an issue that is further upstream than this. Policies, procedures, culture, and resources to execute. None of which are technical. | | |
| ▲ | lorreyfum 2 hours ago | parent [-] | | Not really. Just patch and reboot. Pretty simple. | | |
| ▲ | ImPostingOnHN an hour ago | parent [-] | | can't reboot, the machinists have the windows sized and positioned on screen just like they like them since 1997, so if you reboot it will cause downtime |
|
| |
| ▲ | throwaway894345 7 hours ago | parent | prev [-] | | Kind of feels like national security is the job of the federal government. Seems fair to say the federal government should do their job. They started a war for no reason and failed to anticipate not only these infrastructure breach but also the closure of the Hormuz strait. | | |
| ▲ | pudgywalsh 6 hours ago | parent | next [-] | | So the federal government should be responsible for every rinky-dink water well in Bumblefuck, Minnesota? > failed to anticipate not only these infrastructure breach They've been warning them for close to two decades. Minnesota chose the path of no locks on their front doors and are now crying that someone walked in without knocking first. | | |
| ▲ | jimberlage 4 hours ago | parent | next [-] | | The federal government could potentially oversee the most populous areas of the state and those near a military installation pretty easily. They don’t have to look at every one. | |
| ▲ | throwaway894345 5 hours ago | parent | prev [-] | | The federal government makes sure everyone who sends a venmo for $60 pays taxes on it so yeah I think securing our national infrastructure is not an unreasonable expectation. | | |
| ▲ | zdragnar an hour ago | parent [-] | | There's no "national infrastructure" for water. Aside from what the EPA does, it isn't within the remit of the federal government to manage municipality water systems. | | |
| ▲ | throwaway894345 an hour ago | parent [-] | | My wording was ambiguous. “national infrastructure” can refer to either the infrastructure in our nation or infrastructure managed by the federal government. I meant the former. |
|
|
| |
| ▲ | fathermarz 5 hours ago | parent | prev [-] | | I will repeat a comment from below. There are over 150k water utilities alone in the US. Passing the buck to the Federal Government is not understanding the problem. | | |
| ▲ | throwaway894345 5 hours ago | parent [-] | | There are 150 million taxpayers and the federal government regulates all of them. I don’t see why they can’t audit 0.1% of that. If there are 150,000 utilities then absent some regulation, some of them will fuck up. If we want fewer fuckups, you need regulation. | | |
| ▲ | fathermarz 4 hours ago | parent [-] | | What’s the penalty you would impose on a rural water system that has under 10 employees that services thousands of people for water and/or wastewater? What does that audit look like and how frequent does that happen? It’s a security assessment? A quality assessment? A risk assessment? I’m open to the idea, but I will repeat, I don’t think the problem is well enough understood for a “make the feds do it” type of comment. | | |
| ▲ | throwaway894345 4 hours ago | parent [-] | | It’s the same regulatory/incentive toolbox as any industry, including possibly accepting lower security standards for tiny treatment plants just like we accept less security for podunk airports. > make the feds do it National Security has always been a federal government responsibility. You make it sound like I’m expecting the federal government to take on some new responsibility. If the federal government starts a war with another country they’re absolutely responsible for minimizing by collateral damage at a fucking minimum. | | |
| ▲ | fathermarz 3 hours ago | parent [-] | | National Security has always been a federal government responsibility yes. But what does that fundamentally mean for boots on the ground? NSA doesn’t do IT for the DoD/W, DHS doesn’t do IT for the government, CISA only gives guidance where they can. And IT does not equal OT. The issue comes down to actual skilled people hours to do the work and resource constraints to do so. I agree that in theory this would not be a stretch if the stars aligned, but these are for the most part, not federal government funded entities nor government controlled even at a state level. They are usually clooged together by 100 years of paper maché. And that’s just water. What about Energy? Data Centers? Pharma? Regulation is way too far behind to just instantly drop a silver bullet. And 100% agree that we are witnessing repercussions of leadership that did not have much forethought but that ain’t new and goes back quite a ways especially in CI. | | |
| ▲ | throwaway894345 an hour ago | parent | next [-] | | > The issue comes down to actual skilled people hours to do the work and resource constraints to do so. It comes down to incentives. If you want broad security you have to do more than hope that every water utility will both hire good people and also allow them to do their jobs properly. > But what does that fundamentally mean for boots on the ground? How does any regulation look on the ground? How does the federal government regulate banks and airports? > these are for the most part, not federal government funded entities nor government controlled even at a state level Neither are banks or airports > What about Energy? Data Centers? Pharma? Energy and pharma are already regulated. Maybe data centers will be eventually if they are deemed sufficiently critical. > Regulation is way too far behind to just instantly drop a silver bullet. I don’t know what this even means in the context of securing our water system. Do you mean to say that regulation can’t ensure that these software systems don’t use default passwords and so on? > And 100% agree that we are witnessing repercussions of leadership that did not have much forethought but that ain’t new and goes back quite a ways especially in CI What is new is that we started a war with a country with a respectable technology competency without doing anything to shore up our defenses. | |
| ▲ | ImPostingOnHN an hour ago | parent | prev [-] | | > IT does not equal OT I'll say, you should see the hours I have to work sometimes. Honestly I've rarely seen IT jobs pay OT. |
|
|
|
|
|
|
|
|
| ▲ | dylan604 3 hours ago | parent | prev | next [-] |
| CISA was formed in 2018, so not quite 15 years but closer to half that. The security industry as a whole has been yelling for longer than 15 years about the vulnerability of utilities. They've been marked as soft targets before the Bush administration restructured the government. |
|
| ▲ | throwaway894345 7 hours ago | parent | prev | next [-] |
| Yes, utilities shouldn’t be negligent, but national security is 100% the federal government’s responsibility. If the vulnerabilities were so trivial, then it’s even more damning that the federal government was caught with its pants down, particularly since they were the only ones who knew they would be starting a war. > finger-pointing isn't going to fix it. Your entire comment was finger pointing… |
| |
| ▲ | fathermarz 5 hours ago | parent [-] | | There are over 150k water utilities alone in the US. Passing the buck to the Federal Government is not understanding the problem. | | |
| ▲ | throwaway894345 4 hours ago | parent [-] | | There are 1000x as many tax payers and the government still makes sure every single mom who gets a venmo payment for $60 pays taxes. National security is a federal responsibility, they should absolutely do their jobs. |
|
|
|
| ▲ | idontwantthis 8 hours ago | parent | prev | next [-] |
| Until the people in charge face jailtime for hurting innocent people, why would they care? The government shouldn’t be warning, it should be ordering and imprisoning. And funding and educating where there are genuine gaps. |
|
| ▲ | cyanydeez 7 hours ago | parent | prev | next [-] |
| a broken clock, yada yada. |
|
| ▲ | AnimalMuppet 7 hours ago | parent | prev [-] |
| "I blame it on Minnesota because they are grossly incompetent." "I think Minnesota is behind it." The first quote makes it the state's responsibility to secure local water systems, which I'm not sure that it is. The second makes it at least sound like the state of Minnesota is the entity running the attack on local water systems within their state, which is off in paranoid conspiracy territory. Trump was absolutely wrong. |
