Yep, exactly this. If I have anything with sugar in the house, it'll get consumed in 24-48h. The solution is to just not have anything with sugar in the house.

If I want to splurge with a chocolate or ice cream bar, I take a walk to my corner store and buy just one, and eat it right away. It's extremely cost inefficient compared to if I bought a gallon of ice cream from the store, but that's not what I'm optimizing for here.