What bandwith limits? Im assuming the forward and backward passes have to be done sequentially?
Yes also passing data within each layer