ulimit?

or more probably the c api setrlimit(2)

That does not cap memory per query, which should be implemented by postgres explicitly to capture only the memory that really counts towards "the query" (fine-grained memory limits; ulimit is not that).

Be careful with the setrlimit/ulimit API family, generally it doesn't do what you want. You can limit virtual memory (but... why?) or specific segments like stack, etc. There is also RLIMIT_RSS which sounds like what you'd want, but alas:

    RLIMIT_RSS
        This is a limit (in bytes) on the process's resident set (the number of virtual pages resident in RAM).  This limit has effect only in Linux 2.4.x,  x  <  3 and there affects only calls to madvise(2) specifying MADV_WILLNEED.