ADAM does not work on simple convex problems [1].

  [1] https://parameterfree.com/2020/12/06/neural-network-maybe-evolved-to-make-adam-the-best-optimizer/
  [2] https://arxiv.org/pdf/1905.09997

[1] refers to [2], which shows that ADAM is not as efficient as gradient descent with line search on some problems, including neural networks.

▲ _alternator_ 2 hours ago | parent [-]

I'll point out that "does not work" is not the same as "not as efficient" :) But it does seem the Adam paper had an error.

I think that Nesterov's first order method is the most efficient general first order algorithm on convex problems, so anything else is in some sense worse. (Edit: removed incorrect ADAM comment.)

▲

thesz an hour ago | parent [-]

Yours' "not as efficient" in [2] means that, sometimes, ADAM "does not work." Look at figure 2, ADAM literally does not work in the case of "true model."

	▲	_alternator_ 26 minutes ago \| parent [-]
		Yes, apologies, I didn't read the articles you linked before posting this. I did update the comment. I don't think this changes the point, which is that most optimization methods used in AI owe a substantial intellectual debt to convex optimization theory.