So yeah, this was what I was thinking for a while. What about a more nonlinear estimator? Intuitively seems similar to me.

I believe the reason it works in nonlinear cases is that the derivative is “naturally linear” (to calculate the derivative, you are considering ever smaller regions where the cost function is approximately linear - exactly “how nonlinear” the cost function is elsewhere doesn’t play a role).