Car headlights are regulated. My hunch is that the regulations are based on technical background that is not up to date with modern light sources including LEDs and HIDs.

Styling works against us too. The ability to control the geometry of the light beam improves with the size of the optics relative to the emitter, but people want a car with sexy little lights.

I designed optics for lighting in a past life, though never for an automotive application. This issue is actually on my radar because of the blinding brightness of bike lights on the local bike paths.