Did something similar a while back [1], best way to learn neural nets and backprop. Just using Numpy also makes sure you get the math right without having to deal with higher level frameworks or c++ libraries.

[1] https://github.com/santinic/claudioflow

Its nice! Yeah a lot of the heavy lifting is done by Numpy.