> In other words, with log utility it is optimal to take a 1 in 3 chance of ending human existence in exchange for a 2/3 chance of dramatically raising living standards by a factor of 55.

With no evidence that living standards raise by much if any. "Abundance" is macro hand-waving. Existence of AI is not in isolation from all other factors on earth as well. AI will not increase the quantity of enjoyable housing, will not lower the costs of energy, will slightly temporarily increase calorie productivity of agriculture but not nutritional productivity and will not replenish soil, will not improve environmental degradation of soil, air, water, and so on. We'll drown in more plastic than ever. People might do things to improve abundance but probably won't any more than they do today, for usual reasons of convenience and profit.

A machine that can perfectly selectively weed eliminating the need for herbicides would be significant.