>Unfortunately, finite differences have a big problem: they only compute the derivative of f in one direction. If our input is very high dimensional, computing the full gradient of f becomes computationally infeasible, as we would have to evaluate f for each dimension separately.

And it is terribly unstable numerically. f(x) and f(x+h) are very similar, h is very small. You have to expect destructive cancellation to happen. For black boxes it is the only real alternative though, you can do a bit better by taking a derivative in both directions.