It’s like using a non-cryptographically secure RNG: if you don’t know enough to look for the fsync flag off yourself, it’s unlikely you know enough to evaluate the impact of durability on your application.

It is if they’re in the same physical datacenter. Usually the way this is done is to wait for at least M replicas to fsync, but only require the data to be in memory for the rest. It smooths out the tail latencies, which are quite high for SSDs.

Yes, good call! You can batch up multiple operations into a single call to fsync. You can also tune the number of milliseconds or bytes you're willing to buffer before calling `fsync` to balance latency and throughput. This is how databases like Postgres work by default--see the `commit_delay` option here: https://www.postgresql.org/docs/8.1/runtime-config-wal.html

/dev/null is even faster.

> NATS is very upfront in that the only thing that is guaranteed is the cluster being up.

Not so fast.

Their docs makes some pretty bold claims about JetStream....

They talk about JetStream addressing the "fragility" of other streaming technology.

And "This functionality enables a different quality of service for your NATS messages, and enables fault-tolerant and high-availability configurations."

And one of their big selling-points for JetStream is the whole "stora and replay" thing. Which implies the storage bit should be trustworthy, no ?

Pretty sure SQL Server won't acknowledge a write until its in the WAL (you can go the opposite way and turn on delayed durability though.)

> Filesystems, databases, all kinds of systems do this. They have some hacks to prevent it from corrupting the entire dataset, but lost writes are accepted.

Woah, those are _really_ strong claims. "Lost writes are accepted"? Assuming we are talking about "acknowledged writes", which the article is discussing, I don't think it's true that this is a common default for databases and filesystems. Perhaps databases or K/V stores that are marketed as in-memory caches might have defaults like this, but I'm not familiar with other systems that do.

I'm also getting MongoDB vibes from deciding not to flush except once every two minutes. Even deciding to wait a second would be pretty long, but two minutes? A lot happens in a busy system in 120 seconds...

I think “most people will have to turn on the setting to make things fast at the expense of durability” is a dubious assertion (plenty of system, even high-criticality ones, do not have a very high data rate and thus would not necessarily suffer unduly from e.g. fsync-every-write).

Even if most users do turn out to want “fast_and_dangerous = true”, that’s not a particularly onerous burden to place on users: flip one setting, and hopefully learn from the setting name or the documentation consulted when learning about it that it poses operational risk.

In the defense of PG, for better or worse as far as I know, the 'what is RDBMS default' falls into two categories;

- Read Committed default with MVCC (Oracle, Postgres, Firebird versions with MVCC, I -think- SQLite with WAL falls under this)

- Read committed with write locks one way or another (MSSQL default, SQLite default, Firebird pre MVCC, probably Sybase given MSSQL's lineage...)

I'm not aware of any RDBMS that treats 'serializable' as the default transaction level OOTB (I'd love to learn though!)

....

All of that said, 'Inconsistent read because you don't know RDBMS and did not pay attention to the transaction model' has a very different blame direction than 'We YOLO fsync on a timer to improve throughput'.

If anything it scares me that there's no other tuning options involved such as number of bytes or number of events.

If I get a write-ack from a middleware I expect it to be written one way or another. Not 'It is written within X seconds'.

AFAIK there's no RDBMS that will just 'lose a write' unless the disk happens to be corrupted (or, IDK, maybe someone YOLOing with chaos mode on DB2?)

Core nats is ephemeral. Jetstream is meant to be persisted, and presented as a replacement for kafka

> NATS data is ephemeral in many cases anyhow, so it makes a bit more sense here

Dude ... the guy was testing JetStream.

Which, I quote from the first phrase from the first paragraph on the NATS website:

    NATS has a built-in persistence engine called JetStream which enables messages to be stored and replayed at a later time.

So is MQTT, why bother with NATS then?

NATS was originally made for simple, fast, ephemeral messaging.

The persistence stuff is kinda new and it's not a surprise that there are limitations and bugs.

You should see this report as a good thing, as it will add pressure for improvements.

"PostgreSQL used fsync incorrectly for 20 years"

https://archive.fosdem.org/2019/schedule/event/postgresql_fs...

It did not prevent people from using it. You won't find a database that has the perfect durability, ease of use, performance ect.. It's all about tradeoffs.

do you have a better solution?

as they would say, NATS is a terrible message bus system, but all the others are worse

Interested to know if you found these issues yourself or from a source. Is Kafka any more robust?

This is just a tl;dr of the article with a mean-spirited barb added.

NATS is ephemeral. if you can accept that, then you'll be fine.