I wonder if part of the problem is that starting from electromagnetism doesn't necessarily lead naturally to classic PBR style raytracing; getting there requires fair amount of conceptual leaps, like going from waves to ray optics, or (usually) going from spectrums to rgb triplets. Many things in graphics rendering relies on scenes being well-behaved and relatively simple in a way that might not be obvious when coming from first principles.

It is also interesting to look what you can do if you make your rendering even more physically based. One example is wave_tracer which combines wave optics with path tracing to enable simulating wider range of phenomenons. They have nice theory writeup on their webpage:

https://wavetracer.dev/theory.html