Is it really that significant? We're talking watts difference and over a year maybe 100kWh difference? $35/year?

I don’t recall the exact numbers, and it’s not apples-to-apples but:

We have 4 sticks of DDR3 totalling 24GB. From what I remember, our home server uses about 70kWh/year to drive the memory. Buying 2 sticks of LPDDR4 would bring that down to about 5kWh (?).

At the time, I remember that a new motherboard and a couple of sticks would’ve paid for itself in 3 or 4 years, IIRC.