If you use memory-mapped files, cached pages count towards the resident set size of your process. If you use ordinary file I/O, they don't. That behavior has amusing consequences in HPC clusters that monitor the memory usage of each job and kill them if they use more memory than they requested.