Remix.run Logo
kashnote 2 hours ago

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI.

I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figured out, but a little sympathy goes a long way. I personally wish them the best and hope their on-call people can go back to getting normal amounts of sleep soon.

gogobio 2 hours ago | parent | next [-]

Sympathy? It's a Microsoft company that is being ran with a consistency of a startup in early seed rounds. Their downtime is abhorrent and unacceptable as far as enterprise goes. Their engineers look like absolute amateurs allowing for such low class work it results in their customers experiencing industry leading downtime.

__MatrixMan__ 2 hours ago | parent | next [-]

Sympathy negated by their 2026 Pwnie award: "Lamest Vendor"

https://this.weekinsecurity.com/microsoft-wins-lamest-vendor...

javier2 2 hours ago | parent | prev [-]

and it has been so damn slow for a number of years now, it drives me insane.

sandeepkd 29 minutes ago | parent | prev | next [-]

I think somewhere down the line you have an answer to problem itself. Its for a while that companies are not into the business of sympathies, we should probably not get into that side for the topic.

1. Github has enterprise users who paid for the service, their day job requires Github to be available and working

2. Github has generous free tier which is the one which is exploring a lot more with the AI generated code.

It is a complexity in itself but the traffic should have been separated, the free users should not be allowed to bring down Github for enterprise customers (Just to clarify, I am free user myself). And if they do not have capacity it would have been perfectly fine to push back or throttle new users/repositories.

herpdyderp 2 hours ago | parent | prev | next [-]

> I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers.

I don't think this is commendable at all. I give GitHub a lot of money and I'm tired of it being wasted with downtime.

jjice 2 hours ago | parent [-]

It's a shame that the GitHub org that we use at my job that we pay a lot of money for gets affected the same way my personal nonsense does.

I don't know the architecture or any of that, but I feel like there could be (and it's not like they would've really known this until the last year or two with the massive spike) separate infrastructure for paid users/orgs vs free the same way they make the distinction with enterprise.

I get the massive load changes that they are under over the last two years, but why does a bunch of vibe coded slop take down the same resources that my company pays for every single month and has for years? I imagine properly splitting that out would be an absolute headache and not worthwhile for them vs stabilizing the rest of the service, but damn it sucks when I get blocked at work because GH is down.

tempest_ an hour ago | parent [-]

Our GitlabCE instance has been sitting in the racks for nearly two years with almost 100% uptime running on 10 year old xeons that have long since paid for themselves.

Of course it is not free of all management but for our use case it is working.

There are hiccups with the CI runners from time time but nothing major and we have another machine in another rack that serves as a backup which can be brought up ~< 20 minutes.

I know companies have long since tossed their expertise for hosting their own stuff in favour of SaaS but at some point its hard to beat the up time of a single machine.

Uvix 22 minutes ago | parent [-]

We have the expertise. We don’t have the interest in fighting our internal “security” teams to keep running VMs.

mronetwo 2 hours ago | parent | prev | next [-]

We have a business relationship so no we shouldn’t have any sympathy. They sell a service and they’re failing to provide it.

amelius 2 hours ago | parent [-]

Why? If we have sympathy for Apple then we can have sympathy for Microsoft ...

mirashii 2 hours ago | parent | next [-]

You're the only mention of Apple in this thread, and I don't see what they have to do with it.

pimeys an hour ago | parent | prev [-]

Wait, I don't sympathize Apple at all... Or any other American corporation.

padjo 2 hours ago | parent | prev | next [-]

I don't typically have sympathy for businesses that fail to deliver a service as advertised.

I can have sympathy for the humans caught in the crossfire but only managing one nine of availability on a commercial service is not acceptable.

VCFundedGenYer an hour ago | parent | prev | next [-]

"Leave the big trillion dollar corporation alone" is not the right idea.

jraph an hour ago | parent | prev | next [-]

They did that to themselves, and doubly so.

They decided they needed to capture the whole open source ecosystem by turning open source work into social networking... on a proprietary platform (because open source is great, especially when it's others' software). That was before they joined Microsoft.

And then Microsoft pushed AI everywhere, including on GitHub itself with copilot.

I would have liked if they had left the open source projects alone and didn't create that FOMO for not using them.

I have no sympathy.

kstrauser 37 minutes ago | parent [-]

> And then Microsoft pushed AI everywhere, including on GitHub itself with copilot.

Ding ding ding, we have a winner. I like AI. I work for an AI company. Still, Microsoft aggressively pushed GitHub users toward Copilot. They don't get to do that and complain about increased volume from AI-generated changes.

No Copilot + reasonable operation: the way things were

Copilot + reasonable operation: Well done!

No Copilot + being overwhelmed by AI commits: Sympathy.

Copilot + being overwhelmed by AI commits: "Where did that petard come from that's hoisting us?"

dpz 2 hours ago | parent | prev | next [-]

Maybe for a free account.

But we pay enterprise license and GitHub is a big dependency in our software flow.

If this continues to be a problem as an enterprise product they need to do something. Otherwise theyre are going to to start losing business

niltecedu an hour ago | parent | prev | next [-]

idk man, we pay stupid amounts of money to microsoft, we are an enterprise customer, expecting better availablity compared to my laptop isnt really a high bar.

mcmcmc 34 minutes ago | parent | prev | next [-]

It’s commendable that they let people give them training data for free?

JDups 2 hours ago | parent | prev | next [-]

Constant building with AI is something that they (Microsoft) promote and are heavily invested in.

0xbadcafebee 27 minutes ago | parent | prev | next [-]

Our company pays for GitHub. We're paying for a broken product that stops our work. I don't have sympathy for the companies whom I pay for a product and give me broken shit in return. This is entirely preventable and their own fault.

A restaurant makes pizzas. They suddenly get 100x more popular. They can't make 100x more pizzas. But they are still taking orders from 100x more people. Not only are they not getting enough pizzas delivered that they took orders for, but in their rush to make and deliver more pizzas, they set the kitchen on fire, which makes an even longer wait for pizzas.

When the pizza you ordered doesn't get delivered, do you have sympathy for the restaurant? Or do you tell them to stop taking orders they can't fill and try not to set the kitchen on fire?

Now consider the pizza restaurant has 21 billion dollars in cash, is taking your money, and not giving you pizza.

bamboozled an hour ago | parent | prev | next [-]

No, we don’t.

theideaofcoffee an hour ago | parent | prev [-]

Give me a break. Sympathy? For microsoft? That might have flown when github was like seven people, but they have nearly unlimited resources to make it better. They're just choosing not to. Let's talk contracts and money before we pull the sympathy card.

I used to be on-call in a high-traffic environment where single customers pushed more bits than entire nations. I chose the role. I didn't want people's sympathy, if anything, I wanted them to complain to management.

If it gets too bad they can quit. Maybe that would be for the best, just wear the thing down until it outright fails and no one wants to touch it. One less bullshit service sucking all of the oxygen out.