Why don’t you look at the source code, it’s only 150 lines?

The nesting is limited by using an int as the depth counter. The C standard guarantees that MAX_INT is at least 32767, so that’s a limit on portable nesting depth. Nowadays int is typically 32 or 64 bits, so a much higher limit in typical C implementations.

If I see correctly, the library doesn’t check for overflow, however. This might conceivably be an exploitable vulnerability (and such an overflow would constitute UB).