Remix.run Logo
Incident with Pull Requests, Issues, Git Operations and API Requests(githubstatus.com)
86 points by maxnoe an hour ago | 53 comments
gen220 an hour ago | parent | next [-]

https://isgithubcooked.com

Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.

taintlord223 18 minutes ago | parent | next [-]

The UI of that page is so nice, should build a github competitor.

The user profile / contributions and PR UX is pretty much the entire "hub" product since git is a fully separate offline app.

embedding-shape 15 minutes ago | parent [-]

> The UI of that page is so nice

Is it? Seems a text description of "Make a website outlining 'How cooked GitHub' is with a modern style" to basically any LLM would produce exactly that UI and design, literally nothing of that design a human had any influence on, besides the ones selecting what training data the used LLMs was trained with.

I think most of us who've tried using LLMs for web-design can recognize that style and design at this point, regardless of model actually used.

angrydev 7 minutes ago | parent | next [-]

Compared to near unusable pages that large organizations produce, yes this page is highly effective at conveying information. Who cares how it was produced?

olmo23 8 minutes ago | parent | prev [-]

What really grinds my gears is how easy it is to get better designs out of LLMs. But if you don't ask, you get the default.

EduardoBautista 42 minutes ago | parent | prev | next [-]

May has been filled with critical issues. It seems it's getting worse over time.

pluc 40 minutes ago | parent | prev | next [-]

Name one thing Microsoft didn't run into the ground post-acquisition

robotmaxtron 25 minutes ago | parent | next [-]

hey now, LinkedIn was terrible before Microsoft.

SteveNuts 7 minutes ago | parent [-]

Java or Bedrock edition, and have you tried logging into your EntraID Microsoft Teams for Xbox account lately? Make sure to check the box to keep you logged in!

darkamaul 15 minutes ago | parent | prev | next [-]

I think Minecraft is still in good shape

embedding-shape 5 minutes ago | parent [-]

I wouldn't know, somehow this game I bought maybe 15 years ago is no longer playable for me, my account was supposed to be migrated from Mojang to Microsoft or similar, but then that never happened or something, and trying to login now asks me to contact Microsoft support, which I've tried 3-4 times, never had anyone respond to me so who knows how the game is today? I stopped trying at this point...

Personally, once a game I own is janked from my hands because of organizational decisions, that's the time I'll stop consider the game "in good shape", but I'm sure the people who had to buy the same game a second time still enjoy it.

elzbardico 36 minutes ago | parent | prev [-]

GH was acquired by microsoft some eight years ago. It has been working quite well until recently.

People may have had complaints about functionality, features, commercial issues, but the thing used to at least have a decent uptime until recently.

bsimpson 32 minutes ago | parent | next [-]

It also used to be run as an independent company with access to MS's resources.

Now it's a unit in their AI hype machine.

modriano 16 minutes ago | parent | prev [-]

MSFT was pretty arms length for the first 5-6 years. I was honestly kind of impressed and it made my opinion of MSFT better. But then AI made it too attractive of a target and MSFT couldn't help but make it a place the former CEO wanted to leave (and it has been running headless for about a year now).

It's quite disappointing objectively, but I expected worse from MSFT.

rvz 28 minutes ago | parent | prev [-]

They are already cooked as this has been happening ever since the Microsoft acquisition and it was run to the ground before 2023.

At this point you would get better uptime by just self-hosting your own GitLab, Forgejo or Codeberg instance instead of dealing with Github's unreliablity.

There is no defending them with their clear neglet and carelessness of the platform.

ckorhonen 31 minutes ago | parent | prev | next [-]

This is getting ridiculous. One particularly concerning thing I’m seeing is that pull requests on both the web UI and API aren’t reflecting all commits or branch changes consistently. It would be very easy to merge something without realizing you’re not actually reviewing the full diff.

xnorswap 35 minutes ago | parent | prev | next [-]

Before clicking, I assumed this was going to be a write-up of the one from a few days ago instead of an entirely new incident.

jamdav16 30 minutes ago | parent [-]

I assumed it was the one from yesterday! Silly me.

hansmayer 17 minutes ago | parent | prev | next [-]

It seems before AI eats software, its going to first eat GH and Microsoft.

throwatdem12311 8 minutes ago | parent [-]

they didn’t think the leopard would eat their face!

spaceman_2020 35 minutes ago | parent | prev | next [-]

is it me or ever since AI coding became the norm, there have been way more outages with otherwise reliable services?

I get downtime on Supabase every few weeks. Even Cloudflare. And now Github

hansmayer 15 minutes ago | parent | next [-]

No, it's not just you. It is fairly obvious what's happening - the same old Entshittificators now have a great tool to up the speed of entshitification by 100x - thus these crappy outages every other day.

throwatdem12311 7 minutes ago | parent | prev | next [-]

> is it me or

No, of course not.

julianlam 20 minutes ago | parent | prev [-]

Not just you, but uncertain whether it's due to unreviewed slop going to production, or increased demand due to slop generation.

renehsz 5 minutes ago | parent | prev | next [-]

https://GiveUpGithub.org

throwatdem12311 9 minutes ago | parent | prev | next [-]

Maybe GitHub needs to freeze free repository creation until they get this under control because this is ridiculous.

embedding-shape 2 minutes ago | parent [-]

I mean, if we're talking about "fixing" the symptoms of the downtimes rather than the sources and causes, I guess they could just null route github.com until they have things under control?

Personally, I think they'd have more luck actually attacking the source, what that might be. Somehow I think Microsoft's push for "Every developer only use AI for development, no manual thinking/coding from now on" is the detrimental step, seemingly many companies are still discovering the right approach. Put a freeze to that, and I'm fairly sure you'd see less downtime pretty much immediately, unless all real engineers already left the company, I'm sure I would have at this point.

voidUpdate 40 minutes ago | parent | prev | next [-]

They may have gotten down to only 2 nines on most of their services, but at least the LLM is still running at full power! must increase value for shareholders

abhashanand1501 10 minutes ago | parent | prev | next [-]

as a github user, we are paying for the slow git operations through our github action minutes, if someone from GH is here, will you be compensating for it?

sibidharan an hour ago | parent | prev | next [-]

Are they running paid marketing campaigns for Gitlab ?

ramon156 24 minutes ago | parent | next [-]

Self-hosting forgejo under tailscale + mirroring public repos through GitHub

Has worked wonders for me :)

EduardoBautista 43 minutes ago | parent | prev [-]

I'd consider self-hosting GitHub Enterprise before putting my team through the pain of Gitlab.

ricardbejarano 38 minutes ago | parent [-]

How is it painful to use GitLab? Curious, as a user of both, I find them both nice. I like GitLab CI/CD more than I do GHA, but that's personal preference/bias more than anything objective.

tux3 28 minutes ago | parent | next [-]

Gitlab CI has some tech debt from accumulating geological layers of different ways to do things, but overall it's pretty good, it scales to more complicated setups, and it's not too painful.

Now the best way to use GHA is to do the bare minimum. Put all your CI logic in a script that you can test locally, and just have GHA run your script. Even that is painful. And, somehow, impossible to make secure without having spent 5,000 hours reading all the previous ways people got pwn'd by Github Action's horrendous security model.

My main problem with Gitlab is that after years I still can't find what I'm looking for in the UI. It's always exactly in the third place I look. Otherwise Gitlab has been good. Even self-hosted works pretty well.

EduardoBautista 23 minutes ago | parent | prev [-]

Everything about their UI/UX screams of doing the bare minimum to check off a box on a feature list. It reminds me of Jira.

hansmayer 13 minutes ago | parent | prev | next [-]

Can't they just use one of Satiya's "powerful daily prompts" and ask the - was it "Mico"? - to excrement their way out of these troubles? Ah - you're telling me those powerful prompts were just bullshit for the lazy office cretin who is mainly reading and writing emails throughout the week? They don't really create any new fucking value? No way - I thought CEOs paid tens of millions of dollars each year had real competence justifying such high salaries.

fen4o 34 minutes ago | parent | prev | next [-]

Tried to do a git push - it succeeded after 3 mins. Then I wanted to open a PR and it failed with a 500 error.

Facepalmed and decided that this is it for today.

dzonga 37 minutes ago | parent | prev | next [-]

git is supposed to be decentralized.

maybe it's time to revert back to the central idea of git & not centralize around a particular provider.

for issues - mailing list will do. you can always slap a beautiful ui if you want to or a tui (as is the fad) these days.

actions can also be decentralized via an API spec & webhooks.

cedws an hour ago | parent | prev | next [-]

I'm so done with GitHub.

dist-epoch 23 minutes ago | parent | prev | next [-]

GitHub is not agent scale.

Multiple companies are trying to create new versioning primitives/architectures which can handle machine-level code generation - 1 commit per second per repo.

It's like switching from horse buggies to automobiles, the whole worlds needs re-architecturing to handle the new load.

The age of boutique hand-coding is being replaced by the age of industrial software factories.

swiftcoder a minute ago | parent | next [-]

> new versioning primitives/architectures which can handle machine-level code generation - 1 commit per second per repo.

This is not a particularly novel level of scale. Facebook's mercurial backend had to handle >5,000 developers committing to the singular monorepo long before LLMs were a thing

julianlam 16 minutes ago | parent | prev | next [-]

Why the heck would you want to do this. Using git as your undo chain sounds like a pretty awful thing to do.

skinfaxi 7 minutes ago | parent | prev | next [-]

This seems odd to me. Why would you need to commit every second?

andyjohnson0 7 minutes ago | parent | prev [-]

> GitHub is not agent scale.

Is the scaling issue with git or github?

maxnoe an hour ago | parent | prev | next [-]

GitHub Incident again/

denysvitali an hour ago | parent [-]

At this point we can even stop specifying that it's GitHub...

Hamuko 34 minutes ago | parent | prev | next [-]

Yesterday my CI runs wouldn't even be created because Actions was eating shit, and today my CI runs get created but fail because the API is eating shit. Fun.

looperhacks 40 minutes ago | parent | prev | next [-]

Maybe we should start posting av story when GitHub has been fine for some time instead of posting every incident

drcongo 29 minutes ago | parent | prev | next [-]

For years we had a GitHub status thing in our Slack but I had to remove it about a year ago because the noise got too much, it would be unbearable in 2026.

rvz 40 minutes ago | parent | prev | next [-]

Again?

It was just yesterday [0] that GA was down and another incident today? I am convinced that Copilot and Tay.ai are destroying GitHub and there is no CEO of GitHub to contact.

Now will you please self-host as I said 6 years ago? [1]

[0] https://news.ycombinator.com/item?id=48278635

[1] https://news.ycombinator.com/item?id=22867803

rob 31 minutes ago | parent [-]

You think a Microsoft chatbot from 2016 is destroying GitHub?

rvz 3 minutes ago | parent [-]

At this point, you might as well say that is what's happening at GitHub with the help of GitHub Copilot since nothing has changed and has only gotten worse over time.

Arbortheus 43 minutes ago | parent | prev [-]

Fed up and bored of this