The simplest approach to whitelisting libraries won't work, since the malicious color parser can just call the whitelisted library.

A different idea: Special stack frames such that while that frame is on the stack, certain syscalls are prohibited. These "sandbox frames" could be enabled by default for most library calls, or even used by developers to handle untrusted user input.