A simple burst of memmap + soft fault with 100 or 1000 threads on a normal laptop would tell you that thread contention is real and cache locality gets destroyed. Couple that with pinned threads. You can see the latency increase by increasing thread count. Add to that the motherboard interconnect tax for numa systems. Work stealing is not the way for increasingly many workloads on modern hardware.

Recently we built Dip, our in-house ephemeral + parallel database, and we went with may coroutines + work pinning to the same thread which also nicely becomes numa aware via architecture.

Increasing threads beyond system's hardware cores/threads resulted only in marginal gains of a couple of milliseconds worth of differences on huge workload with large increase in memory (thread stack) used by the massive number of threads.

▲

rkangel a day ago | parent | next [-]

The Erlang thing is different though - that's a userspace scheduler, scheduling userspace tasks onto one-thread-per-core. Classic M:N scheduling in a similar way to go.

It works extremely well at scale - you can handle 10k connections on a normal machine with very little thought, and WhatsApp has reported handling 1 million. Yes everything has some point at which it won't scale, and apparently that approach (or at least that implementation) struggles at 100 cores. That's not a normal machine.

	▲	elendilm a day ago \| parent [-]
		It appears you are unfamiliar with "may" coroutines which is a userspace M:N scheduler. Erlang phenomenon mentioned is the same and the behavior is easily reproducible on a normal laptop. "may" defaults to the same number of OS threads as there are cores. Scale the set_pool_capacity() to 1000, 5000, or beyond for coroutine scaling. Also try set_workers() for OS threads. Try it any which way and you see thread contention and cache locality penalty increasing latency.

▲

FridgeSeal a day ago | parent | prev [-]

> Increasing threads beyond system's hardware cores/threads resulted only in marginal gains of a couple of milliseconds worth of differences on huge workload with large increase in memory

Careful, if you say that too loudly, the "get rid of async just spawn more threads!!!!!" people will come out of the woodwork to yell at you about how _all_ async is a lie and we should instead pretend none of it exists and just spawn more threads.

▲

elendilm a day ago | parent [-]

I am now trembling at the thought of warriors who will skin me alive :).

Jokes aside, there are use cases for rayon, use cases for tokio async, use cases for "may" coroutines, use cases for a custom scheduling policy, or use cases for a combination of these.

We went with "may" coroutines (with its thread pinning) + custom numa work pinning (to a may thread) due to "may's" lightweight nature and not having to have our functions colorized. We use rayon where cache locality penalty is minor compared to the millisecond latency gains due to work stealing - but this is only for an edge case.

Our log broker Monolog (akin to kafka) which sits atop Dip uses tokio async because it plays nicely with zmq while "may" doesn't.

So I am a big fan of using the right tools for the job. Each technique has its own pros and cons. Informed decisions based on use cases matter above any dogma.

▲

FridgeSeal a day ago | parent [-]

It sounds like you’ve got a super interesting stack going on there, evidently with a large performance/latency focus.

May (haha) I ask what this is in service of? I’m somewhat a fan of the thread-per-core model, so I’m curious as to what you’re doing with it.

	▲	elendilm a day ago \| parent [-]
		Our stack consists of two consumer facing mobile apps, Slyp and SlypBusiness, which necessitated developing the underlying infrastructure over time for scaling and efficiency. That stack includes Dip (our database substrate), Monolog (our log broker), Singularity (our KV engine), Craft (a mobile app for app development using flow charts, snippets, and a WYSIWYG designer), and Portal (a mobile app for remote container management and a HITL AI workflow). Craft and Portal are MVP-style functional, but aren't in production yet. Everything else is being rolled out gradually. You're more than welcome. Happy to discuss any of it in more detail. Rather than digress further in this thread, feel free to reach out privately if you'd like to continue the conversation.