>We could be teaching notes to children objectively like how we teach colors, but we're not.

Do you mean trying to teach all children perfect pitch even though society has no expectation of that? Unlike knowing at least your primary colors which is expected of everyone. I suspect that could be unnecessarily stressful for many.

Or do you mean as some kind of metaphor or analogy? If the latter, I think it would be quite confusing as there are aspects of vision and hearing that are quite different. Pitch classes have no analog in vision that I can think of. Color vision is roughly 3 dimensional but sound is not. The aspects of timbre don't map to color.

I think that understanding music theory does require work. It emerges from physics and physiology and a very long history including a bunch of culturally specific things. Did your ancestors make music with long skinny strings or pipes with nice integer-ish overtones? Here are some tuning systems for you (among them the set of C-D-E-F-G-A-B you mentioned). Did they use bells or gongs with decidedly non-integer ratio overtones? Here's a set of different systems for you!

Anyway, if you have a mapping/analog/metaphor you think is useful between music or sound and color I would be interested to hear it / see it!