Moore's law or one of its generalizations still holds, so it will only be a short matter of time before a $1k computer will be able to train and run a powerful enough model.

I thought Moore's Law came to an end in the last decade?

Certainly the transistors/chip or transistors/$ or flops/$ have not been progressing at the same exponential rate as during 1970-2010. There is still progress, but it's rather slower.