Have you measured this, or is this just an opinion?
Look into /proc/<PID>/status and /proc/<PID>/smaps