If you're bypassing the page cache, what invalidates the page cache so that the next read (from the filesystem) isn't stale?
we also use direct-io for reads.
You have your own read cache then? You're nearly at the Oracle-level of writing your own filesystem at this point.