It turns out that you don't want integers the same size as a pointer because somebody might squirrel away capability bits in your pointer type (see CHERI) and you definitely do not want integers with capability bits.

Rust originally says that its types usize and isize are the same size as pointers, but this was ret-conned in later Rust to say actually they're the same size as addresses for this reason.