As a counter-anecdote, I've had far more trouble over the years swapping physical SIMs than eSIMs. You'd think that going between two phones that use the same size card would just work, but in practice that isn't (wasn't?) always the case.

Never saw an issue moving a SIM from one phone to another (living in Asia, Europe and the US). However last week I got a Airalo E-Sim and apparently it's not possible to transfer it to my new phone.