This stuff exists and it's fairly well-studied. It's surprisingly hard to find without coming across it in literature though, the universe of image processing is huge. Joint demosaicing, for example, is a decades-old technique [0] fairly common in astrophotography. Commercial photographers simply never cared or asked for it, and so the tools intended for them didn't bother either. You'd find more of it in things like scientific ISP and robotics.

[0] https://doi.org/10.1145/2980179.2982399

I trawled through much of the research but as you’ve mentioned it seems to be known only in astrophotography and mobile devices or other similarly constrained hardware.