We've found DuckDB 2.0 to be way faster on high core count machines on EC2. Not just reading data from S3, but the actual computations too. Much more likely to see 100% utilisation up to about 128vCPUs or 192vCPUs, and that CPU usage to translate into workloads completing faster.

Still don't usually see a speedup past about 192vCPU, but prior to 2.0, speed would top out at more like 64vCPU, so this is a big win.

That has enabled us to scale Splink (FOSS record linakge software) to process 1bn records with 10bn record comparisons in 10 mins, which as far as I'm aware makes it the fastest linkage engine of this kind. Most of this performance comes from simply delegating the big calculations to the DuckDB engine! https://moj-analytical-services.github.io/splink/topic_guide...

Thanks to the team for all their work!

▲

gopalv 2 hours ago | parent | next [-]

> Not just reading data from S3, but the actual computations too.

Async io generally causes a CPU bump as a side-effect, when the bottleneck moves from the disk to the CPU (more usually memory bandwidth).

My experience is from a different SQL engine, but the improvement in scan rates directly translates all other operators getting more work to do per unit time.

More CPUs pegged at 100% is always a good sign of smooth flows, but optimizing further requires data layout changes from that point. Even if it is a sub-linear speedup out of concurrency scaling - more with more is still a win, assuming you can pay for more (the Dy-skew paper has an example I can quote).

▲

RobinL an hour ago | parent [-]

Yep that makes sense.

One thing we've seen with DuckDB, which they also mention in their docs, is that it's possible to 'over parallelize'. Beyond some number of cores, the overall workflow measured in wall time actually slows down when you add more CPUs, because of the overhead of parallelizing, despite them running at 100%.

We observe this even if we pull the data from S3 onto the local SSD before processing,

	▲	gh02t 38 minutes ago \| parent [-]
		This is also a problem in classical HPC, there aren't very many algorithms that can scale linearly with cores. That's partially why so many HPC algorithms tend to be built on matrix multiplication and Monte Carlo, because those do tend to scale near arbitrarily.

▲

dicey 3 hours ago | parent | prev | next [-]

I’m a huge fan of splink - and how you’ve been able to adopt duckdb.

Do you find that with the performance of duckdb, you see large deployments of spark based workflows less often? It seems hard to justify the complexity of spark if you can use duckdb , but I’m wondering if you see end users still reach for spark for any reasons?

	▲	RobinL 3 hours ago \| parent [-]
		Appreciate the kind words. There are a variety of people still use Spark, but this is usually due to constraints on their platform/infrastructure, not because it performs better. For instance people on Databricks can sometimes find it hard to use DuckDB on large machines, or they're just more familiar with Spark. For a new user who has flexibility over where to run their workloads, we'd always recommend DuckDB. This is particularly the case now that: - Splink 5 (released in September) allows you to run your workload in n chunks, meaning you can distribute it across multiple machines even on DuckDB - DuckDB 2.0 parallelises better on large machines, so throwing a single massive machine at problems is now more viable In general, after using Spark for a long time, I think for most workloads, it's just not really needed any more. Too complex, too difficult to tune. Perhaps if you have genuinely massive data. I've written a bit more about why I advise against Spark in most instances here: https://www.robinlinacre.com/recommend_duckdb/

▲

4 hours ago | parent | prev [-]

[deleted]