Remix.run Logo
jjice 2 hours ago

> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory).

I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain.

You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org.

[0] https://damrnelson.github.io/github-historical-uptime/

twistedpair 2 hours ago | parent | next [-]

They _are_ the LLM coding agent vendor, and are _owned_ by MS, the ~majority~ biggest shareholder of OpenAI. How can you NOT consider that 20x+ scaling in your capacity roadmap projections, where you are trying to get everyone to use these agents as part of your core OKRs?

Or, maybe your #1 IT priority was moving everything to Azure instead ;)

rafram 2 hours ago | parent [-]

> the majority owner of OpenAI

Largest shareholder (27%), not majority.

chilipepperhott an hour ago | parent [-]

Plurality should have been the operative word.

debarshri 2 hours ago | parent | prev | next [-]

It looks like right after Microsoft acquisition this started going off.

This is the outcome of violating single responsibility principle in business.

Topfi 2 hours ago | parent | prev | next [-]

Github being under the CoreAI division probably also doesn't help the engineers prioritize addressing infrastructure issues and makes using LLM load as an excuse feel self-inflicted. Akin to feeling sorry when a pyromaniacs house burns down...

chrisjj 2 hours ago | parent | prev [-]

They've had a couple of months and still have not brought in sufficient extra capacity?? What's their excuse?