You could go a long way by specifying the reflectiveness over the visible light spectrum and then do the same for light sources. That way, you could do GDR style truly monochrome sodium vapor lamps, which makes telling colors apart impossible.

To do neon colors, you would need some sort of frequency redirect in your texture data as in "this pixel absorbs 80% infrared, but reemits it as visible red". Then you could have black light light sources and cool poster textures as well.

I think doing textures in this format would take too much space, but doing a 256 color palette that way might be fine.

“ultraviolet”, not infrared. IR to visible light would be upconversion.

https://en.wikipedia.org/wiki/Photon_upconversion