Appreciate the kind words.

There are a variety of people still use Spark, but this is usually due to constraints on their platform/infrastructure, not because it performs better. For instance people on Databricks can sometimes find it hard to use DuckDB on large machines, or they're just more familiar with Spark.

For a new user who has flexibility over where to run their workloads, we'd always recommend DuckDB. This is particularly the case now that: - Splink 5 (released in September) allows you to run your workload in n chunks, meaning you can distribute it across multiple machines even on DuckDB - DuckDB 2.0 parallelises better on large machines, so throwing a single massive machine at problems is now more viable

In general, after using Spark for a long time, I think for most workloads, it's just not really needed any more. Too complex, too difficult to tune. Perhaps if you have genuinely massive data. I've written a bit more about why I advise against Spark in most instances here: https://www.robinlinacre.com/recommend_duckdb/