Remix clone Hacker News

This is probably because C NIFs run in the same process as the Erlang scheduler. If you have a long-running or blocking NIF, it can starve the scheduler and cause significant performance degradation across the system.

▲

natrys 6 days ago | parent | next [-]

I think they now have "dirty" NIFs that use a separate scheduler for this.

	▲	throwawaymaths 6 days ago \| parent [-]
		yes, but there is a finite number of them, by default equal to the number of available cores. If your connection stays in c-land for too long you might run into trouble, if more than one connection are desired.

▲

hinkley 6 days ago | parent | prev [-]

I wonder if there's space for a libuv inspired solution now.

▲

toast0 5 days ago | parent [-]

libuv is more or less abstraction around an event loop for async i/o right?

The BEAM is also more or less an abstraction around an event loop for async i/o. If you want async i/o in nifs, I think you want to integrate with BEAM's event loop. Inside NIFs, I think you want to use enif_select [1] (and friends), available since OTP 20 originally from 2017-06-21. In a port driver, you'd use driver_select [2] which I think has been around forever --- there's mentions of changes in R13 which I think was mostly release 2009-11-20 (that may have been R13B though).

[1] https://www.erlang.org/doc/apps/erts/erl_nif.html#enif_selec...

[2] https://www.erlang.org/doc/apps/erts/erl_driver.html#driver_...

▲

hinkley 5 days ago | parent [-]

It uses different threads in order to make infrequent blocking calls look like they are asynchronous.

When we (or at least some quantity of “we”) want is infrequent native calls to be able to fail without taking the BEAM down.

The problem with doing it with threads though is that a bad thread can still vomit all over working memory, still causing a panic even if it itself doesn’t panic.

▲

toast0 5 days ago | parent [-]

Oh I see. If you need to isolate your native code from your BEAM, then you've got several options.

a) run the native code as a separate process; either a port program, a c-node, or just a regular program that you interact with via some interprocess communication (sockets, pipes, signals, a shared filesystem, shared memory segments if you're brave)

b) some sort of sandybox thing; like compile to wasm and then jit back to (hopefully) safe native.

c) just run the native code, it's probably fine, hopefully. My experience with NIFs is that they are usually very short code that should be easy to review, so this isn't as bad as it sounds...

If your native code is short, option c is probably fine; if your native code is long, option a makes more sense. If you want to make things harder without real justification, b sounds good :P

▲

hinkley 5 days ago | parent [-]

I suspect the confusion of priorities comes from running native code that’s time consuming and wanting it to horizontally scale with your cluster, which is easier if you try to let the beam do it.

While what is easier for those of us not working on the beam is to put the glitchy code into its own service and put up with the maintenance overhead.

But when you have one or two solutions the friction to move to three becomes difficult. People start talking about how having dozens will be chaos. While true, sometimes you really do just need three and it’s not a slippery slope.

	▲	toast0 4 days ago \| parent [-]
		> sometimes you really do just need three If it's worth doing, it's worth doing three times.