How long ago was this, or can you share more context about data and mem size you experienced this with?

DuckDB has introduced spilling to disk and some other tweaks since a good year now: https://duckdb.org/2024/07/09/memory-management

▲

wewewedxfgdf 7 days ago | parent [-]

3 days ago.

The final straw was an index which generated fine on MacOS and failed on Linux - exact same code.

Machine had plenty of RAM.

The thing is, it is really the responsibility of the application to regulate its behavior based on available memory. Crashing out just should not be an option but that's the way DuckDB is built.

	▲	alex-korr 6 days ago \| parent [-]
		I had the same experience - everything runs great on an AWS Linux EC2 with 32GB of memory, same workload in a docker on ECS with 32GB allocated gets an OOM. But for smaller workloads, DuckDB is fantastic... however, there's a certain point when Spark or Snowflake start to make more sense.