Wouldn't that lead to interference with the original device using that MAC address? Ie both of you getting packets meant for the other etc?

Yes. It’s a bad (maybe even illegal?) thing to do. The victim device will start losing packets, and it’ll look to them like the network is just unreliable. Eventually they give up and disconnect, and you take full control.