| ▲ | cletus 35 minutes ago | |
My qualifications for talking about notifications. I wrote what I believe to still be the current notification system for Facebook Live, which peaked at >10M QPS and averaged somewhere in the 1-8M QPS range. This isn't exactly the same as notifying someone about their order and shipping but it shares a lot of the same problems. Now none of this sent notifications directly using Google's or Apple's notification services. There were whole levels of infrastructure for that at (then) Facebook. Also, there were types of notification where an ML recommendation had to decide if a given notification was something you might be interested in ie it was a decision to send or not send. Notifications are inherently asynchronous and you need to immediately get out of the mindset of verifying if they were sent. You don't actually care if they're sent immediately or an hour from now generally. There may be valid reasons for waiting that have nothing to do with your process. For example, when a new product goes live you'll get a flood of orders. That's going to slow things down and you'll absolutely need to throttle your notifications or you'll kill downstream services. Disclaimer: I had to deal with the ML Infra teams yelling at us for doing exactly this. Retry logic isn't something you should concern yourself with. You need a service you can rely on to send a notification. That's it's responsibility, not yours. If you successfully pass it on to that service, your job is done. It's up to that service to schedule, send and retry if necessary. I wouldn't concern myself with quiet hours. Users silence their phones. They'll figure it out. That's just unnecessary complexity and it's opaque. People will likely get confused why they're not getting an email or notification for something they expect. Even with a relatively small number of users this is guaranteed to happen. Don't try and get too clever. The author brings up a good point about separating kinds of notifications. This is something I did. Example: FB at the time had a setting on a page where you could say that you absolutely wanted a notification when the page went live. I separated those into a completely different process to the "maybe you're interested" process. There were a lot fewer of those so they went out really quickly. News organizations and the like would use those to sanity-check that their live was working and it made them feel better to immediately see a notification rather than having to wait 5-10 minutes. Plus, the user has shown the intent that they want those notifications so we should prioritize them. The talk about mass sends is what I was referring to earlier although in our case, it was a page going live that had 20M followers. And the spike here was news organizations tended to go live at the top of the hour. That was extremely noticeable. Users would have preferences (eg no live notifications) and some users were more likely to be interested than not (eg if they'd watched previous lives by that page or any lives at all). I'm not sure "nightmare" is the word I'd use but it's definitely complicated. The first step is getting out of the mindset of getting a send result and doing retries yourself. | ||