The way you get out of a local maximum is to force yourself to try something radically new every so often, even if what you were doing before was working just fine.

This is the 'stochastic' part of 'stochastic gradient descent,' and it's as important for human minds as it is for ANNs. These math wizards seem to be stuck in a rut of their own digging.