The layers don’t have to be non-linear, but you need a non-linear activation function between them. People often overlook the importance of the network topology and the activation functions. The weights alone are not a complete description of the network.

Yep.