When you call brk() from a user program you use the lib wrapper, and that does indeed return 0 on success and -1 on failure. But the actual Linux system call, which I presume is what musl calls, returns the current break on failure instead of -1.

See the NOTES section in man brk(2):

> C library/kernel differences

> The return value described above for brk() is the behavior provided by the glibc wrapper function for the Linux brk() system call. (On most other implementations, the return value from brk() is the same; this return value was also specified in SUSv2.) However, the actual Linux system call returns the new program break on success. On failure, the system call returns the current break. The glibc wrapper function does some work (i.e., checks whether the new break is less than addr) to provide the 0 and -1 return values described above.