Remix.run Logo
zehaeva an hour ago

I don't think this portents anything great for software in general.

We're a good year+ into the use LLMs for all major bits of software that we all rely upon and GitHub here is down to one 9 of uptime. I've been using GitHub for a _long_ time, my first commits there go back to August 2009!, and I honestly don't recall GitHub going down as much as it has in the last year.

I'm sure there's other things happening in the background, but I can not help but believe that this is directly correlated with the increase of LLM usage.

Though I would love to hear someone else's pet theory how a rock of the internet went from four+ nines of uptime to maybe one.

cortesoft 43 minutes ago | parent | next [-]

Are these outages caused by introduced bugs, though, or by load issues?

As someone who has spent many years working in high load environments, this is not an uncommon pattern.

You design a system and it works great. It can handle failures, load spikes, it is horizontally scalable, things are great. You think you figured it out.

And then load keeps increasing and you suddenly hit a tipping point where everything keeps failing, and you cant keep up. The things that you thought were perfectly horizontally scalable turn out to have a bottleneck you didn’t even think about until you got to a truly massive scale. Your systems suddenly don’t have the excess capacity to handle load spikes or catchup work, so suddenly any failure cascades and recovery is more and more difficult. You can’t solve the problem with additional hardware, and your perfect scalable design actually can’t scale any more.

This doesn’t have to be about GitHub using LLMs in their code to still be related to LLMs. GitHub gets a lot more commits now because of LLMs and probably get a lot more reads because of LLMs as well.

It could be that the extra usage just pushed them past one of those capacity thresholds.

askonomm an hour ago | parent | prev | next [-]

To me this correlates more to them being bought by Microsoft, a company known for being seemingly incapable of creating quality software to the point that it's not even funny anymore, and also known for sloppifying all the products they touch.

IshKebab an hour ago | parent | next [-]

I don't think so. GitHub was bought by Microsoft 8 years ago and people have only started complaining about its uptime in the last year or so - exactly correlating with the surge in LLM use.

axod 35 minutes ago | parent | next [-]

https://damrnelson.github.io/github-historical-uptime/

Seems pretty conclusive. Very similar story when they bought skype.

onraglanroad an hour ago | parent | prev | next [-]

I don't think so. I've seen those complaints for more than a year.

I have an Ops background and I strongly suspect they were given a stupid timeline for the Azure migration.

I've got to believe Microsoft have decent Ops people but the management wanted to move faster than was reasonable and screwed it up. Move one thing at a time and double check it all works and you can do a migration like this.

miyoji 26 minutes ago | parent | prev | next [-]

> people have only started complaining about its uptime in the last year or so

I'm sorry but this made me laugh out loud. That isn't true at all, this has been going on for years. This conversation[0] from six years ago has discussion about the outages starting to become much more frequent in December 2019. It has never gotten better in that time, it's just continually degraded.

[0] https://news.ycombinator.com/item?id=22935941

pluralmonad an hour ago | parent | prev [-]

Didn't github also migrate at azure recently? That certainly can't help.

shevy-java 28 minutes ago | parent | prev [-]

It's true, Microsoft made github worse, but the more recent issues seem to have to do a lot more with Microsoft selling its soul to AI. Microsoft really appears to have gotten dumber as they became dependent on AI. Most recent example: they used to promote Win11 and 32GB RAM. Now they are down to 8GB silently ... this is quite hilarious. There are now so many side effects that you see degradation in so many other areas. Or the gaming industry: it is not quite dying but it is taking a huge hit with skyrocketing RAM prices. Consoles selling less is an example here. It's quite fascinating how deadly disruptive AI is now.

tylerdavis 26 minutes ago | parent | prev | next [-]

They mentioned earlier this year that they were beginning the migration to Azure and that it would take a couple years. I would assume it has more to do with that migration then anything else.

denysvitali 35 minutes ago | parent | prev | next [-]

Microsoft acquisition which forced to migrate all to Azure.

zehaeva 8 minutes ago | parent [-]

I can totally buy that this is a larger contributor! Thank you, I wasn't aware that it was going on right now.

gershy an hour ago | parent | prev | next [-]

I guess github is kind of a shared garden. Interesting that, like in game theory, if everyone is using it too much, no one gets to use it.

BigTuna an hour ago | parent | prev | next [-]

It's probably not attributable to AI in the way that you're thinking - Github has been absorbing an exponential increase in usage, and that increase is mostly due to new AI-related projects being created and worked on.

Though I'm sure some of the blame can go to internal slop code.

porridgeraisin an hour ago | parent | prev [-]

For the longest time, I didn't appreciate the "AI-induced traffic" excuse. But seriously, I checked the rough github egress for our lab versus an old log from 2024, and there's an order of magnitude or two difference. From asking around, it seems people all have the gh cli tool and let it loose with parallel tool calls and e.g LLM's polling Actions in a background bash while loop with sleep $TOO_FEW_SECONDS. Some people use a variety of skills where the agent makes a commit every few code changes, and uses Issues for its memory/log. And they have O(5) sessions at the same time doing all kinds of crap. It's the same with PR checks/PRs. Recently we also saw continued usage throughout the night as well, which did not exist pre coding agents. Loops or whatever they call cron jobs in the harnesses these days is the reason. It must be adding up.