Who didn't? Depending on exactly how you interpret the notion of "inventing backpropagation" it's been invented, forgotten, re-invented, forgotten again, re-re-invented, etc, about 7 or 8 times. And no, I don't have specific citations in front of me, but I will say that a lot of interesting bits about the history of the development of neural networks (including backpropagation) can be found in the book Talking Nets: An Oral History of Neural Networks[1].

[1]: https://www.amazon.com/Talking-Nets-History-Neural-Networks/...

▲

catgary 3 days ago | parent | next [-]

I think it’s the move towards GPU-based computing is probably more significant - the constraints put in place by GPU programming (no branching, try not to update tensors in place, etc) sync up with the constraints put in place by differentiable programming.

Once people had a sufficiently compelling reason to write differentiable code, the frameworks around differentiable programming (theano, tensorflow, torch, JAX) picked up a lot of steam.

▲

convolvatron 3 days ago | parent | prev | next [-]

don't undergrad adaptive filters count?

https://en.wikipedia.org/wiki/Adaptive_filter

doesn't need a differentiation of the forward term, but if you squint it looks pretty close

▲

albertzeyer 3 days ago | parent | prev [-]

How do you not have the citations in front of you? They are all in the article? I don't expect that any relevant (re)invention of backprop is missing there. Or, if you really know some reinvention of backprop that is not mentioned here, tell Jürgen Schmidhuber, he is actually very curious to learn about other such instances that he is not aware of yet.

▲

mindcrime 3 days ago | parent [-]

They are all in the article?

Maybe they are. I'm not here to do a deep research project that involves reading every citation in that article. If it makes you feel better, pretend that what I said was instead:

"I don't have all the relevant citations stored in my short-term memory right this second and I am not interested in writing a lengthy thesis to satisfy pedantic navel-gazers on HN."

Or, if you really know some reinvention of backprop that is not mentioned here,

WTF are you on about? I never made any such claim, or anything remotely close to it.

▲

albertzeyer 3 days ago | parent [-]

I thought that is what you mean when you said "a lot of interesting bits about the history of the development of neural networks (including backpropagation) can be found in the book Talking Nets", that there is some relevant reference to backprop which is missing here in the linked article.

I don't really understand your negativity here, and what you are reading into my comment. I never asked you to do a research project? I just thought you might know some other references which are not in the article. If you don't, fine.

Note that I don't expect that any relevant reference is missing here. Schmidhuber always try to be very careful to be very complete and exhaustive cite everything there is on some topic. That is why I was double curious about the possibility that sth is missing, and what it could be.

	▲	mindcrime 3 days ago \| parent [-]
		I thought that is what you mean when you said "a lot of interesting bits about the history of the development of neural networks (including backpropagation) can be found in the book Talking Nets", that there is some relevant reference to backprop which is missing here in the linked article. Nah, I wasn't trying to imply that that book had anything more than the article, at least in regards to the back-prob question specifically. Just pointing it out as one more good resource for this kind of historical perspective. I don't really understand your negativity here, and what you are reading into my comment. I never asked you to do a research project? I just thought you might know some other references which are not in the article. If you don't, fine. No worries. I may be reacting more to a general HN meme than to you in particular. There's a certain brand of pedantry and obsessive nit-picking that is all too common here IMO. It grates on my nerves, so if I ever seem a little salty, it's probably because I thought somebody was doing that thing. It's all good. My apologies for the argumentative tone earlier. Schmidhuber always try to be very careful to be very complete and exhaustive cite everything there is on some topic. Agreed. That's one reason I don't get why people are always busting on Jurgen. For the most part, it seems that he can back up the claims he makes, and then some. I've heard plenty of people complain about him, but I'm not sure any of them have ever been able to state any particular sense in which he is actually wrong about anything. :-)