| ▲ | vlovich123 4 hours ago |
| Not sure what you mean. Nothing about preadv lets you indicate you only want to read what's already in the page cache. And io_uring and preadv aren't orthogonal - you can give io_uring a preadv op to do the scattered read instead of issuing separate read OPs although I'm not 100% sure how much of a win that is in practice. Also, I think you misunderstood the blog as it's describing application read-ahead which is what you have to do when using O_DIRECT. |
|
| ▲ | marginalia_nu 4 hours ago | parent | next [-] |
| So I have a buffer pool with O_DIRECT reads. I implement read-ahead in the application by (optionally) preadv:ing a single read into multiple destination buffers in the pool, leaving them unpinned, since as long as you aren't up against the bandwidth limit of the drive, a larger read is generally as fast as multiple smaller one on modern hardware. I've tried doing this with io_uring as well, but found just eating the preadv syscall cost was faster. |
| |
| ▲ | vlovich123 an hour ago | parent | next [-] | | Do this with io_uring with the preadv syscall. It’ll be the same or faster (faster only if you can do something else while waiting for I/O or you can submit multiple requests simultaneously - a single io_uring will be basically identical) | | |
| ▲ | marginalia_nu 41 minutes ago | parent [-] | | io_uring is significantly faster for certain workloads, but you can't expect that merely running a syscall via io_uring will somehow magically make it faster. Replacing single sycalls with their equivalents in io_uring is generally slower than just making the syscall directly. io_uring still uses syscalls after all. io_uring generally only wins if you can amortize its overhead across multiple simultaneous operations. Implementing readahead would be such a case, except you can accomplish the same amortization with a single preadv instead, which again turns into a single syscall for multiple reads. |
| |
| ▲ | lossolo 3 hours ago | parent | prev [-] | | > a larger read is generally as fast as multiple smaller one on modern hardware. Not always if by modern you mean NVMe drives.
One synchronous preadv() for 256 KiB gives the kernel/device one big request but 16 independent asynchronous 16 KiB reads can be serviced concurrently.
So the latter gives the NVMe controller 16 operations it can schedule in parallel.
So depending on the workload and hardware, offsets, filesystem and request sizes that can give you lower aggregate latency or higher throughput. | | |
| ▲ | marginalia_nu 2 hours ago | parent | next [-] | | If you submit 16 contiguous read requests the system will just merge them into one large read request. Modern SSDs tolerate moderate queue depths very well, but piling on the I/O queue also incurs tail latency jitter unless you're able to ensure the queue depth stays in the moderate range and never goes higher. All else being equal, fewer larger requests is better for I/O latency (though read amplification for the sake of reading more data obviously doesn't help anyone). Though in this scenario, we're mostly comparing the syscall overhead of a single preadv against io_uring bookkeeping for multiple preads, regardless of how you submit the reads they end up being the same operation. | | |
| ▲ | vlovich123 an hour ago | parent [-] | | Doesn’t the Linux kernel automatically merge requests for contiguous regions into a single request? I’m not sure 16 read I/Os submitted to io_uring for a contiguous 64kib region will behave differently at the disk level than a single 64kib request. There’s a little bit extra work the kernel has to do to merge the requests but that should be negligible. In a poorly written micro benchmark test it’s possible the kernel will fail to coalesce all of them because your submissions aren’t visible all at once (ie it starts submitting requests and doesn’t have an opportunity to merge). Whether that actually is possible to happen requires digging a bit more into the Linux kernel source. | | |
| ▲ | marginalia_nu 37 minutes ago | parent [-] | | Yes that is correct. You can look at 'rrqm/s' and '%rrqm' in `iostat -x` to get statistics for how many requests are merged, and `rareq-sz` for the average request size. |
|
| |
| ▲ | ComputerGuru 2 hours ago | parent | prev [-] | | I think that’s only true on paper; in practice it’ll be true only when you have competing reads (at a thousand-foot view) and it might be possible to algorithmically bundle a portion thereof. It originally let SCSI controllers attached to spinning rust HDDs optimize physical manipulation of the disk heads to optimized queued reads of data in a “traveling salesman” sort of way, but modern nand flash can only internally read a full page at a time (which may be much greater than even the apparent physical sector size) anyway and with a strictly constant cost regardless of the “physical location” of the data on the non-existent platter. Old drives had optimization constraints like higher sequential read speeds at the outside of the platter (more bytes per physical rotation) and extremely pathological cases for data written to the innermost tracks of the platter. Individual requests were much finer-grained and the latency was much more varied, so a request from app/thread X for as little as 512 bytes from one location could be cheaply piggy-backed on an existing request from app/thread Y to read multiple megabytes from a physically proximate source that would otherwise have seriously delayed or starved the queued waiting read while the outstanding request was serviced. In fact, one consistently sees higher bulk IO numbers when using physical media that has been formatted with a large sector size compared to the old 512 byte fixed emulated size. You’d routinely see lower latency and higher IOPs with 4kn (HDDs or SSDs) than you would with 512e disks, even with SCSI or AHCI controllers that featured similar pipelining support to today’s NVME controllers (or even if you place a spinning rust HDD behind NVMe today!). | | |
| ▲ | vlovich123 44 minutes ago | parent | next [-] | | Mostly true but as someone else pointed out it’s still better to issue a single large multi-page contiguous read than the same read broken down into separate requests because the I/O queue isn’t infinite. So both the Linux kernel and the SSD microcontroller still benefit from merging contiguous requests on their end. However, if you issue it correctly at the application level that’s still going to be better in that you’re never going to encounter a situation where you accidentally don’t get the desired coalescing. | |
| ▲ | RossBencina 2 hours ago | parent | prev [-] | | > disk heads to optimized queued reads of data in a “traveling salesman” sort of way Luckily the read heads only have one degree of freedom, so the "elevator algorithm" is sufficient: https://en.wikipedia.org/wiki/Elevator_algorithm |
|
|
|
|
| ▲ | wahern an hour ago | parent | prev [-] |
| > Nothing about preadv lets you indicate you only want to read what's already in the page cache. preadv2 + RWF_NOWAIT |