| ▲ | mulmboy 2 hours ago | |
People are usually surprised to hear that polars can be slower for fairly pedestrian operations, especially with smaller datasets. For example take a 5000 x 3 dataframe of float64 and sort by one column and you'll find polars takes about 2.5x as long. If you set POLARS_MAX_THREADS to 1 then it's faster. Though this all depends on the machine. Polars tends to shine with larger datasets or where it can heavily take advantage of query planning. Don't skip your profiling | ||
| ▲ | winwang an hour ago | parent [-] | |
(Without profiling or looking into this at all) I'd guess this has to with thread creation, inter-core communication/latency, and possibly having to merge results or otherwise interleave operations. SMT is another likely candidate. Regardless, CPUs are really good at single-thread. | ||