Be careful with the setrlimit/ulimit API family, generally it doesn't do what you want. You can limit virtual memory (but... why?) or specific segments like stack, etc. There is also RLIMIT_RSS which sounds like what you'd want, but alas:

    RLIMIT_RSS
        This is a limit (in bytes) on the process's resident set (the number of virtual pages resident in RAM).  This limit has effect only in Linux 2.4.x,  x  <  3 and there affects only calls to madvise(2) specifying MADV_WILLNEED.