Don't you think that we will eventually get more specialized hardware that will greatly improve efficiency? Running neural networks on GPU:s seem like quite wasteful?

we will, but data centers will compete for that hardware with retail consumers