Program instructions size is small thus loading is fast, so no need to worry about that too much. I'd look on different things first.
Have you measured this, or is this just an opinion?
Look into /proc/<PID>/status and /proc/<PID>/smaps