What prevents an attacker from using nested maps to make the server spend exponential amounts of CPU and memory on the response? Is there some kind of limit on the total number of response items?

The application should track resource use and implement limits as needed.

I know that sounds like a cop-out, but this is really true of any protocol, and the RPC protocol itself has no real knowledge of the cost of each operation or how much memory is held, so can't really enforce limits automatically.