| ▲ | pletnes 2 hours ago | |||||||
What is the benefit of using async io for csv and parquet files? | ||||||||
| ▲ | skeeter2020 an hour ago | parent | next [-] | |||||||
Helps use all of the available bandwidth by not blocking (otherwise idle) worker threads waiting for data to arrive. Requests and processing are managed independently, so you essentially get better utilization & improved performance. | ||||||||
| ▲ | jtbaker an hour ago | parent | prev [-] | |||||||
highest impact is when there is network overhead, like accessing them over S3/HTTP/S, and you're accessing many files as part of a query, like using a glob `-*.parquet` pattern. I imagine the same applies when you're doing local I/O, but not as pronounced of an improvement due to lower latency. | ||||||||
| ||||||||