> energy efficient lights seem to have no downsides

Efficiency is good, but this isn't true. Decades into LED-ification we still suffer shit-tier PWM strobing, poor dimming, disruption to wildlife and human life, and streetlights turning purple en masse.

(The last two are installation and quality issues, but when driven by lowest-bidder compliance with simplistic "efficiency" mandates that's what you get).

Lower color rendering index than incandescent lights too. High CRI LEDs are available, but they're still worse than incandescents and less efficient than the conventional LEDs.