> rgb values still points to a single color, which maps back to a single value on a 0 -> 1 or red -> violet continuum.

No, it doesn't. Wavelength is unidimensional, but color can mix many wavelengths, and RGB is a 3d color system which doesn't cover all combinations of visible light but does approximate the way most human vision works, and is therefore useful as a description for human-perceived colors (and more accurate than picking a single point on the unidimensional wavelength spectrum for that purpose.)