I wonder, is there a form of analysis which lets you quantify how ambiguous a set of symbols is? Maybe related to entropy?

Obviously one symbol can mean literally anything, but you could also have very long strings of symbols with many different meanings.

Yes. Somewhere in Claude Shannon's work, called the "unicity distance".