> The company is wildly popular and any service launched is going to have a lot of traffic and a lot of attacks right away.

Doesn't this also inform you that your cache will be very large, so you shouldn't use growable structures with slack space when cache entries won't grow; slop space reduces the size of your cache. And also that the query volume will be high so the cached data should require as little work as possible before returning data; spending time marshalling response data on every cache hit increases response time and decreases capacity.

▲

sophacles 20 hours ago | parent [-]

RAM is cheap. I'd find myself far far more concerned with:

* unbounded growth of the cache and properly invalidating after TTL expires (a few GBs of slop is nothing on a server with 64 or more GBs of ram, unbounded growth is a problem).

* making sure the DNS implementation works correctly on both the serving side and recursive resolution side.

* What strategy is best for deduping recursive requests across machines (if something a few miliseconds away has a live result, why do a full lookup taking hundreds or thousands of milliseconds?). This potentially improves RAM usage across the datacenter too from not having a given record on dozens (or more) machines' local cache. I don't know exactly how they do it, but naively I'd look at some sort of DHT shaped solution to look for records in peers within the datacenter. Or maybe some sort of tiered caching with the upper tier being sharded on domain name or the like.

* The biggest performance gains cloudflare can provide in Web and DNS cache come from a cache hit. This is on the order of 10s or 100s of ms due to having a big cache and short distance to the requesting machine. A suboptimal lookup algorithm that is a few microseconds slower in local compute and ram access is just not as important as the other concerns for dedup and cache sharing. That's not to say it's unimportant, just that it's not the top priority when you're trying to deliver this much larger performance gains from other aspects of the system. Thats why they are getting to it several years after release.

Cloudflare writes a lot about distributed systems solutions to various problems. They likely don't think as hard about single machine performance as much as whole datacenter performance when approaching problems.

Keep in mind that the per-server cost of the whole program pre-optimization seems to be about 10GB (from the graph in the post). IME that's not bad for a big busy caching service.

▲

toast0 19 hours ago | parent [-]

> The biggest performance gains cloudflare can provide in Web and DNS cache come from a cache hit.

Using twice as much ram per cache entry makes the cache half as large, assuming your cache is bounded by ram, unless the queried, unexpired result set is less than the ram budget (which I would tend to doubt... lots of randomized queries out there; maybe I'm wrong if the cache size dropped).

When you're storing billions of records, it makes sense to spend a few minutes to consider how they're used and make a good choice about how to store them.

When you're getting a cache hit tons of times per second, it makes sense to consider every step and which ones don't need to happen every time. You have to consider every step while you're pursing correctness anyway, so might as well have the performance lens active too.

I'm not asking for heroic optimization: I didn't ask for vectorized stuff or kernel/nic offloading or kernel bypass networking... Just you have to use some data structures, you might as well not use ones that are expensive for features you don't need; and you have to store something in your cache, you may as well store something that requires less munging on the way out.

If this were a small local cache, that didn't want to use something already existing like unbound for some reason then yeah, data structures don't make a huge difference, extra marshalling doesn't make a huge difference, just don't reimplement all the CVEs that BIND had in the 90s. But if you're going to allocate 100 TB of ram, make it count. Even if you do use twice the ram but you get value from it, maybe that's fine... I've run wacky systems with bloated storage when there was a benefit. Vec doesn't give any value over a Box<[]> in this case; convenience or lazyness would be fine except that the sheer number of objects makes it worth the few minutes it takes to do something better.

	▲	sophacles 11 hours ago \| parent [-]
		> Using twice as much ram per cache entry makes the cache half as large, assuming your cache is bounded by ram, unless the queried, unexpired result set is less than the ram budget (which I would tend to doubt... lots of randomized queries out there; maybe I'm wrong if the cache size dropped). This is true. I'm arguing that its unlikely this was ever bound by available RAM. Cloudflare is a DDoS protection company that absorbs attacks. They have a lot of available capacity at any moment. When you're building a service in a sitaution where you have more capacity than you'll likely need. The savings were 100 TB across >300 data centers. The savings were on the order of 50%. So prior to this reduction the service was using something less than 2/3 of TB per datacenter. The service ram usage was about 10GB per instance according to the graph in post. IDK how cloudflare divides thier stuff between machines, but assuming they don't run less than 64 GB per server that's less than 12 servers per datacenter of ram for a flagship product, and they likely run it spread across 65 of the machines in the datacenter that are also doing other stuff. The per-instance RAM likely isn't the the concerning limit. Overall RAM usage is proabably a bigger concern. Thats why I would think about dedup between instances and distributed caching strategy first. I could focus on redudcing the ram needed per service instance and get a 50% reduction per machine. Or I could focus on deduping 1/n (where n > 2) reduction in total memory usage across all instances. Personally if I was worried about reducing RAM I'd put more energy into growing N. However all this is a red herring. The assumption people are making is that the cache was always read-only, and it's obvious that Box<[T]> was the best decision because in a RO cache smaller entries hold more things. The 1.1.1.1 service advertises improved DNS performace. That's its value add. The biggest performance gain you can have from a cache is not having a cache miss, and in DNS a cache miss means a very expensive recursive lookup. So there's concerns about how to minimize those lookups. If one instance has does a lookup, it makes sense to share that result to the other instances that may need to do a lookup [1]. I don't know off the top of my head if it makes sense to get those updates and modify the existing record or just replace it in the local cache. That comes down to locking strategies and reading patterns in the specific code and service traffic patterns. Until i have hard evidence one way or another I'd like my cache to be able to do both and keep the data structs modifiable until that's nailed down. If per-isntance ram ever becomes the issue, there's easy wins there to buy me time to find better large scale solutions to the problem. No one is disagreeing that the larger datastructures are larger. No one is disagreeing that they take more RAM, and or even if RAM was the the problem reducing it would be good. The thing people are pointing out is that this isn't a homework problem about an optimal cache structure in a vacuum. We're pointing our that engineering real large scale solutions has a lot more to consider than a homework problem, and that the thing you're harping about likely didn't have any real budgetary or noticable performance impact on bulding that system. The reduction in ram is just a smallish improvement in operating costs after all the more expensive stuff was figured out. Put another way 100TB of RAM is ~$350K. Thats one engineer year for a mid-level engineer.[2] Would you rather spend that money to save an equivalent amount of money somewhere, or... would you spend that money putting the engineer on something that saved $700K elsewhere (alternately that generated $700K)? [1] I talked a lot about dedup and the simple gotcha is "hahah then its not deduped so you need smaller objects". But on a service that is running on a few dozen instances having a few redundant copies to deal with loss of a machine and/or load can still result in 1/(n>2) savings in total ram. [2] I'm not saying someone worked on this for a year btw, a couple people likely spent a couple months on the code, validation and testing of it. A manager spent time overseeing it. Operations people spent time understaning any effects it had on running systems. Costs add up and it wouldn't suprise me if this didn't end up being roughly break-even for the year.