Asynchronous I/O in DuckDB: Work, Thread, Work

bburnett44 2 hours ago | parent | next [-]

Using 512gb of ram for a 22gb remote file does feel a bit weird for a benchmark but maybe they couldn’t get a large number of cores without lots of memory?

	▲	otterley 2 hours ago \| parent [-]
		Most cloud providers start with a 2:1 ratio of memory in GiB to CPU cores and go up from there. Databases also are the most common workload for large-memory systems because they benefit so much from large buffer caches.

▲

myshapeprotocol 5 hours ago | parent | prev | next [-]

Deep dive into asynchronous I/O architectures like this is pure engineering gold for high-performance data processing. Excellent breakdown.

▲

pjot 2 hours ago | parent | prev | next [-]

I wonder how this would work in trying to parallelize the worker threads (multiple duckdb instances) coordinating them via Quack.

Ducks all the way down!

▲

datadrivenangel 3 hours ago | parent | prev | next [-]

DuckDB is trending towards becoming a query engine, specifically the fastest analytical query engine. This is very good.

▲

abofh 3 hours ago | parent | prev | next [-]

Do they have SSL updates yet? Signing is great, but using https means not fighting firewalls to start a job

▲

mansi1010 5 hours ago | parent | prev | next [-]

This is such a long waited feature!

▲

myshapeprotocol 40 minutes ago | parent | prev [-]

Brilliant deep dive into asynchronous I/O and execution thread models. Essential reading for high-performance data engineering.