To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside.

If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health

That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.

> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory).

I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain.

You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org.

[0] https://damrnelson.github.io/github-historical-uptime/

For the amount of money enterprises are paying to GitHub, there is a reasonable (and contractual) expectation of uptime. I don't think my boss would find it to fun if I bailed work to hit the gym just because GitHub was down.

> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

Which means they have a pricing problem. Rate limit non-paying users.

Stop blaming LLMs: https://damrnelson.github.io/github-historical-uptime/

Actually, I do have an important fix for a deal we are trying to close that does need to go out. And I am paying GitHub to host this. Once or twice, I can see, but GitHub's SLA is getting worse than just hosting it myself, and that's the entire reason I pay GitHub.

Geez, sorry that the paid service my company paid for is down and I can't do my work.

This has to be rage bait, this is a critical piece of infrastructure for many people. Stuff going down can lead to deployments failing and as you mentioned in many cases emergency hotfix's.

The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.

> due to LLM sloppers pushing large amounts of trash code

Which they encouraged by pushing Copilot down everyone's throat

GH shouldn't have promoted LLM usage the way they have if they didn't have the infrastructure to support it. Regardless of how you feel about the code quality (which you have zero evidence of), if GH has made commitments to supporting broad LLM usage, they need to back that up with the proper hardware and without whatever fragile SDLC processes they have.

> To everyone who is angry: calm down. Github’s servers are constantly on fire

Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?

What if we're not angry and just perplexed? Do we calm down further or can we still ask questions?

Hard disagree, people are paying GitHub a lot of money.

If they are facing an influx of commits from bots maybe they need to start metering commits from those accounts and charge them money.

This is a self imposed problem.

Why do you get to tell me how I should feel?

what if I do have an emergency hotfix for a production outage though ;-;

It's every other day now. We pay for a service, we expect service.

This might be naïve but wouldn't the appropriate response be to reduce access/rate limit free/new accounts in order for service to be maintained for everyone else?

> as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

Github and their parent company are active participants in pushing LLM-driven coding.

I understand the need to level-head the discussion and calming everyone.

However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow.

The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available.

(I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)

Allowing and accepting the LLM load is a willful choice they made and are making at the expense of their users, including their paying enterprise users.

They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.

As a _paying_ customer, I disagree.

> constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

I find it hard to believe that github APIs don't have rate limits to handle traffic spikes.

Alternatively, don't use GH for critical infrastructure they are no longer reliable, simple as.

Well, they’re a critical piece of infrastructure with terrible stability. I’m in the process of migrating us off GitHub now.

I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.

Or, you know, it's perfectly reasonable and natural to feel angry when a service you paid for gets worse over time.

> Unless you have a emergency hotfix (you don’t)

Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.

While I agree in principle, Github with their CI is very critical to many parties.

Why would you take what they say at face value? I believe that this is basically a lie covering up deeper rot.

No one ever has to commit anything time sensitive.

> increased something like 50x due to LLM

This should not be an excuse since they probably can use AI to fix it /partially sarcasm

> due to LLM sloppers pushing large amounts of trash code

hey bud your bias is showing

it's been trash for years, their status page just doesn't reflect that.

You know, I am not employed.

I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket.

So now I am not angry because some critical piece of US-american infrastructure is down all the time.

I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.

Wouldn't be surprised if we lost free private repos because of the slopocalypse. Though most don't seem to be shy about sharing their slop with the world so I'm not sure if that will actually increase stability.

In this case, perhaps they shouldn't allow AI slop or promote usage of AI agents...

> go hit the gym or walk outside.

My mind immediately went to this classic xkcd: https://xkcd.com/303/

FYI this is an argument to not use Github, not an argument to use it.

It has nothing to do with load.

What if you, you know, have a job?

is the infrastructure of the free tier shared with that of paid users? That might be the issue. Otherwise if usage of paid users scales, then it would be fine.

Agreed. This impatience culture is so toxic. Been down less than 30 mins.