Remix.run Logo
the_mitsuhiko 6 hours ago

I'm not working at this company but I found that these types of problems can often be simplified in the architecture.

> Most meetings start on the hour, some on the half, but most on the full. It sounds obvious to say it aloud, but the implication of this has rippled through our entire media processing infrastructure.

When you can control when it happens, you can often jitter things. For instance the naive approach of rate limiting users down to quantized times (eg: the minute, the hour, etc.) leads to every client coming back at the same time. The solution there is to apply a stable jitter so different clients get different resets.

That pattern does not go all that well with meetings as they need to happen when they happen, which is going to be mostly the hour and 30 minutes etc. However often the lead up time to those meetings is quite long, so you can do the work needed that should happen on the hour, quite a bit ahead of time and then apply the changes in one large batch on the minute.

You have similar problems quite often with things like weekly update emails. At scale it can take you a lot of time to prepare all the updates, often more than 12 hours. But you don't want the mails to come in at different times of the day so you really need to get the reports prepared and then send them out when ready.

rsanek 4 hours ago | parent [-]

They mention that they implemented jitter later in the post.

the_mitsuhiko 3 hours ago | parent [-]

But my reading of their jitter is a very narrow one for the actual connection to the database. They are still doing most of the work on the minute.