Perfectly demonstrating the truth of the "Microsoft org chart" cartoon.

https://bonkersworld.net/organizational-charts

It is also company behind NuGet.

Guess what they did a year ago.

They removed 700 or so packages from NuGet proactively but those turned out to be false positives.

It is hard to do the right things.

i mean, then you say it like that…

For those not getting the joke, GitHub has had an increasingly difficult keeping itself up since Microsoft acquired them.

It's gotten a lot worse (and made news) more recently, as the downtime as increased.

I really hope this pushes users (here: devs and maintainers) to decrease their reliance on Microsoft and especially stop outsourcing security to them.

Migrate off vscode already.

Not holding my breath. This issue has been open since 2018 https://github.com/microsoft/vscode/issues/52116

GitHub confirmed that it's indeed the nx console extension, in their blog post: https://github.blog/security/investigating-unauthorized-acce...

It feels so bad to see the "You need go give Chrome SUID Root for the sandbox to work". Setting a Web Browser SUID Root was an old joke about clueless users. It was the worst security screwup someone could imagine.

Don't build your ide on electron then.

podman seems to handle rootless namespaces just fine, minor caveat for some perf overhead but it's not the end of the world.

As others have said it's just a fraction. I'm in a medium size tech-related company and we have 7500+ in one Github org. We have two orgs, so altogether easily 10K+. Of course most of it is stale, obsolete, sandbox, personal tools, etc. I wouldn't be surprised if Github would have 100K+ internal repos or even more.

Uber had 8000 repos at one point with 2000 engineers - https://highscalability.com/lessons-learned-from-scaling-ube...

I worked for a food retail store once. I remember going in the first day wondering, how hard can it really be... From the outside, it looks like they have a simple website. The website to order things on was an amalgamation of 300+ repo's. GitHub lost less in this breach. It takes a lot of effort to keep things simple as you grow.

Something cool that I've always liked about working at GitHub is how much of the company _runs on GitHub_ -- A lot of teams, even non-technical teams, have their own repos just to organize docs/SOP's/designs/etc like a traditional knowledge work company might use a Sharepoint

Personally I have over a hundred, especially from quick prototypes, studies or instances of templates so I can easily see how over 18 years and many hundreds of employees you end up with thousands.

3800 is low for an org like GitHub. Glad it’s highly likely not all their repos are compromised.

I was part of an org with more than 15k repos

It sounds low to me, I worked at a Fortune high number a few years ago and they had more.

Am I missing the joke here... they have hundreds of millions of repos.

In my personal experience, give it a decade or two, and any corporation will accumulate hundreds (or even thousands) of abandoned internal repos containing discontinued services, POCs/prototypes that never went anywhere, etc – people forget to archive them, or aren't sure whether something is still in use or not so err on the safe side.

AI is making this even worse. With coding agents, anyone can throw together a quick internal prototype of any idea they have, even if it has no hope of ever making it to production.

really? I mean these are internal repos. Probably most of them are random one-off experiments or a place to park code. Google has 2,900 "public" repos on github. Microsoft has ~8k "public" on github too. Can't even imagine how many they have on their internal systems.

They have 800 engineers. So 3,800 repos is high, but not crazy.

Some of those could be forks.

The problem is that the main target for these repos are the internal IaaS type repos that contain much of the juicy information.

A fine grained token is likely to have read access to the IaaS repo as that is likely the very repo they are operating on when the malware compromises them.

3800 repos up for blackmail may make a good headline but it's likely that Github don't really care about 3798 of those repos being made public. It'd be annoying for those 3798 to be made public but they can deal with that. It's the 2 repos that contain really important stuff that they really don't want to be made public. You can't rely on fine grained tokens to limit the leak of these things as, at some point, someone with that very access will get compromised.

Limiting TTL on tokens/auth isn't a perfect solution either. If the token is leaked via some malware it can be used to clone repos within minutes (even seconds) of being leaked. No-one wants to have to perform 2FA every few seconds in order to get on with their day.

IP based restrictions may help, but then the malware would probably evolve to include a tailscale/wireguard key so that the clone/exfiltration is done from the existing IP address and then the data is proxied away separately.

Future dev environments are going to be heavily sandboxed in terms of "do github stuff in this sandbox, copy files to another sandbox to do package updates, vet everything coming back, etc"

I was noodling around with personal access tokens today on GitHub and found out that you actually can restrict tokens to specific repositories, orgs, etc. Not sure if actions is a scope that is available or not.

It's working, we just don't know what it's doing.

No time to deploy, nobody must move a muscle during the forensic investigation!

Don't jinx it!

I am so, so stressed about Sublime Text... It feels like a massive disaster just waiting to happen. They don't even run their own package marketplace :(

> but apparently Microsoft has put way more effort into their Copilot slop than security.

Your security or their money (selling Copilot to enterprise customers): what would they choose, hmm? Surprise!

Why would you sandbox extension?

Just don’t install crap maybe.

A few days ago I saw I had an update to the Twig extension. The UI flagged it as having new executable code in the update bundle, so I didn't install the update, disabled the extension as I wasn't working on Drupal views that day, and went about my work. I didn't have time to investigate the new update's contents. When I went back to the extension page, it was taken down: https://open-vsx.org/extension/whatwedo/twig

I'm not saying it was whatwedo.twig, but I'm not saying it wasn't, either.

Edit: If anyone's got a good recommendation for a twig formatter for Cursor / VS Code, please let me know.

There are rumours that was NX Console VS code extension

https://github.com/nrwl/nx-console/security/advisories/GHSA-...

https://www.stepsecurity.io/blog/nx-console-vs-code-extensio...

The security measure that the developer didn't use was completely refusing to use vscode.

vscode has no security model. It's not like swiss cheese where there are holes and some of the go all the way through. vscode is all hole with some cheese on the side. There is absolutely no isolation between the front-end process, the backend size (the thing that runs in the remote or the devcontainer), and any extensions or anything that might be in a repository whose authors you "trust".

(We merged this thread hither - it was originally in https://news.ycombinator.com/item?id=48201316)

The 3800 repos weren't exfiltrated from the compromised machine.

The malware (be it a VSCode plugin, an npm package, or whatever is next) simply slurps up all of the users private keys/tokens/env-vars it can find and sends this off somewhere covertly.

It's trivial to do this in a way to avoid detection. The small payload can be encrypted (so it can't be pattern matched) and then the destination can be one of millions of already compromised websites found via a google search and made to look like a small upload (it could even be chunked and uploaded via query parameters in a HTTP GET request).

The hackers receive the bundle of compromised tokens/keys and go look at what they give access to. Most of the time it's going to be someone's boring home network and a couple of public or private github repos. But every once in a while it's a developer who works at a big organisation (e.g. Github) with access to lots of private repos.

The hackers can then use the keys to clone all of the internal/private repos for that organisation that the compromised keys have access to. Some organisations may have alerts setup for this, but by the time they fire or are actioned upon the data will probably be downloaded. There's no re-auth or 2FA required for "git clone" in most organisations.

With this data the hackers have further options:

a) attempt to extort the company to pay a ransom on the promise of deleting the data

b) look for more access/keys/etc buried somewhere in the downloaded repos and see what else they can find with those

c) publish it for shits and giggles

d) try and make changes to further propagate the malware via similar or new attack vectors

e) analyse what has been downloaded to work out future attack vectors on the product itself

Right now Github (and others recently compromised in similar ways) will be thinking about what information is in those internal repos and what damage would it cause if that information became public, or what that information could be used to find out further down the line.

"Customer data should not be in a github repo" is all well and good, but if the customer data is actually stored in a database somewhere in AWS and there's even just one read-only access token stored somewhere in one private github repo, then there's a chance that the hackers will find that and exfiltrate the customer data that way.

Preventing the breach is hard. There will always be someone in an org who downloads and installs something on their dev machine that they shouldn't, or uses their dev machine for personal browsing, or playing games, or the company dev infra relies on something that is a known attack vector (like npm).

Preventing the exfiltration is virtually impossible. If you have a machine with access to the Internet and allow people to use a browser to google things then small payloads of data can be exfiltrated trivially. (I used to work somewhere where the dev network was air-gapped. The only way to get things onto it was typing it in, floppy or QIC-150 tape - in the days before USB memory sticks.)

Detecting the breach is nigh on impossible if the keys are not used egregiously. Sure some companies can limit access to things like Github to specific IPs, but it wouldn't take much for the malware to do something to work around this. (I can see things like a wireguard/tailscale client being embedded in malware to allow the compromised machine to be used as a proxy in such cases.)

Alerting that requires manual response is nigh on useless as by the time someone has been paged about something the horse has already bolted.

Knowing what has been taken is also a huge burden. 3800 repos that people now have to think about and decide what the implications are. Having been through something like this in the past there are plenty of times people go "I know that repo, it's fine, we can ignore that one" only for it to contain something they don't realise could be important.

These kind of attacks are going to become increasingly common as they're proven to work well and the mitigations for them are HARD. It doesn't need to be targeted at all either, you just infect a bunch of different things and see what gets sent in.

If companies continue to not pay the ransom then we're going to get a lot more things published and many companies having to apologise for all manner of things that end up being leaked.

friendly reminder: use vim :)

It's definitely AI slop. So tired of pushing AI-generated crap to production at my company

The data has been stolen by a criminal group. Paying for "restoring" the data does not guarantee they will delete all copies. There is no way of proving they actually did and they have in fact very little incentive to actually delete it.

You have to take their words for it but how can you trust crooks?

> this is not a ransom … Send your offers … we are not interested in under 50k…

It is a blind auction with a $50k minimum bid.