I'll point out that "does not work" is not the same as "not as efficient" :) But it does seem the Adam paper had an error.

I think that Nesterov's first order method is the most efficient general first order algorithm on convex problems, so anything else is in some sense worse. (Edit: removed incorrect ADAM comment.)

▲

thesz 4 hours ago | parent [-]

Yours' "not as efficient" in [2] means that, sometimes, ADAM "does not work." Look at figure 2, ADAM literally does not work in the case of "true model."

	▲	_alternator_ 4 hours ago \| parent [-]
		Yes, apologies, I didn't read the articles you linked before posting this. I did update the comment. I don't think this changes the point, which is that most optimization methods used in AI owe a substantial intellectual debt to convex optimization theory.