Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago.

(The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)

In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.

I'll add more as we find out more, but it probably won't be till later this afternoon PST.

Crazy that Dang literally manages HN in his sleep!

We all knew that but I haven't seen any confirmation before this.

Hey dang, don't worry. It's just a site for reading articles and reacting to them.

Enjoy your deserved sleep and if for a couple of hours it's down, so be it.

Thanks for your continued service!

I was personally worried if there was some major outage of the whole world or something the first time hackernews didnt work because I didnt expect hackernews to go down but rather, something even more catastrophic than aws going down must happen (because we see major cloud outage posts)

https://downforeveryoneorjustme.com/hacker-news

This website had many instances of reports, the last I saw were 52 reports in only a short frame of time, the maximum reports on this are 118 it seems.

> In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - at that point I was just sleeping through it.

Its okay I suppose, have you figured out who is crawling hackernews so much tho, was it a ddos attack or an AI company trying to get data, doesn't hackernews support an api and I am sure that there are datasets for it too so Its interesting why they might crawl but we all know the reasons why as they have been discussed here.

Maybe it would be fine if ops alerts were silenced during normal US sleeping hours?

HN is important, but unlikely much harm could be done before morning.

(Source: Lost a lot of sleep at one place, enough to realize that sleep interruption and deficit has significant costs.)

No apology needed. We all needed to stop procrastinating anyways :)

During the last week my IP was banned for unknown reason. Glad to hear it might not be a problem from my side.

I’d love to know more about what running a site like HN involves, would be great to get a write up of what it’s like running something like this at this scale (and what kind of traffic you guys get)!

The transparency is deeply appreciated by me and others. We don't pay to keep HN on, so we cannot complain. Thank you and the rest of the team for all you do to give us a corner of the internet that is quite 'different' from the rest of the wild west that is the web.

Frankly, I don't understand why someone would even try to crawl Hacker News.

There is an official dump which doesn't even require parsing HTML at all: https://console.cloud.google.com/marketplace/details/y-combi...

In a situation like this one, good crisis leadership is essential. dang, HN will help you with tips from vast collected experience (please chip in):

1. Blame: The first thing to do is to point the finger. That doesn't mean analysing the technical issue, which can delay this step and limit your options, but figuring out who is politically easiest to blame. Often, that's the new guy, but outside contractors and vendors without good connections are also a common solution. Even if you are technically responsible for hiring them, you can always push them under the bus with a little skill. This small sacrifice helps unify, focus, and motivate the rest of the team.

2. Emotion: Inject your emotion into the situation and make that the implicit, but indisputable priority. Particularly, outrage and anger - This is completely _____. These people are utterly _____ (I'd use all caps, but that's not allowed on HN). Make sure everyone's attention is over their shoulder, on your emotion, and infect the team with it. Threats are an effective tool here - this is a crisis, and anyone who is calm is not emotionally engaged. Otherwise, they won't care enough about this problem - without you driving them, they probably wouldn't care much at all. Anyway, you don't have time for niceties like empathy or even basic respect.

3. Speed: Respnsiveness to stakeholders is very important. People need answers now. Give them answers they want to hear, outcomes they will be comfortable with. Don't worry if different groups hear different things. Your team will find a way to make it all work - that's their job.

4. Communication: Good communication is essential. Make sure you clearly tell your team what they should be doing; repeat it several times to prevent misunderstanding. Especially people with experience can have minds of their own; keep them on track. The situation is a crisis so you can't take any risks; stay on top of them and everything they do, and give input if you're not certain they are doing exactly what you would be doing.

5. Victimhood: Find a way to turn the tables: Make it about you, and how you're the victim here, and feed the fire with more outrage. With this and outrage, nobody will undermine the team by challenging your ideas or authority, which is the most essential component of a successful outcome. Remember, without you this all falls apart.

Have I missed anything?

Short lived and driven by good intentions– all's good. Thanks again for keeping this thing going!

Even after providing firebase endpoint, crawlers come to the site ?

> anti-crawler protections

Sometimes I could not open the comment section, receiving a blank page with "... We're sorry" or something along these lines when opening from new private window. It works when opening normally.

Logging in on the private window seems to resolve the issue. Can you take a look on this if possible?

Can't speak for others, but I'm sure i'll be pretty fine if no one gets woken up if HN is down...

Of course, they'd better restore service after they wake up naturally, because I need my HN dose. But it's not worth losing sleep over it.

> the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users

Was the blocking returning “Sorry.” instead of any page content? A couple of days ago there was a few hours where when I’d go to HN I could load the main page as a non-logged in user. But if I tried to log in I would get “Sorry.” instead. I also got the sorry message if I tried to click on user profiles of other people and a few other pages.

I am assuming that the reason I could see the front page itself and discussions on posts on the front page is that they were in a shared cache for non-logged in users, but that when I clicked on some pages like some random user pages those were not in cache and hit the origin server and it blocked those with “Sorry.” like it did for log-in attempts.

I also tried to go to the unblock IP page, but that one also returned “Sorry.”

For a while I was scratching my head wondering if I had gotten some malware on one of my computers that was aggressively making requests to HN, and that I had become IP banned because of that. Since I think my actual request rate from browsing and commenting should be pretty average. I read HN a lot, but not that much :p

Later in the day, or the next day, things were back to normal and I could log in again. Presumably after those anti-crawler protections had been relaxed again.

> The reason I did that is that the anti-crawler protections also unfortunately hit some legit users

How does this happen?

Looking forward to the post mortem. :)

dang - just to say, we've all done it...

Just out of curiosity, if HN is still running on one physical system, what does a daily or weekly traffic chart look like for the switch port facing it?

I didn't realize you were carrying the pager too! Kudos!

Decades ago I had to write a Perl script to auth to the site for proper downtime checking. Some things never change :) Good luck with the triage.

dang!