Yeah it's complicated, and that's the thing about parsing anything, the more complicated and unpredictable the input and the harder it is to parse.

Does it need to be human readable, does it need to work across all platforms. Does the it need to be secure. These things change everything. Speed, reliability, security pick one.

Your point about being compliant with the real spec is the difference between a 20 line scannf and and a 1000 line function. Yeah. Ha.