Least squares is guaranteed to be convex [0]. At least for linear fit functions there is only one minimum and gradient descent is guaranteed to take you there (and you can solve it with a simple matrix inversion, which doesn't even require iteration).

Intuitively this is because a multidimensional parabola looks like a bowl, so it's easy to find the bottom. For higher powers the shape can be more complicated and have multiple minima.

But I guess these arguments are more about making the problem easy to solve. There could be applications where higher powers are worth the extra difficulty. You have to think about what you're trying to optimize.

[0] https://math.stackexchange.com/questions/483339/proof-of-con...