Another possibility is that a third factor is causing both better living conditions and lower fertility, not that better living conditions inherently cause lower fertility.

I believe lower fertility is most closely associated with education for women. Women with an education sometimes find interests other than being a baby factory.