I cut the difference in speed by half by taking the activations on the GPU. Time to sleep but will continue tomorrow.