Neither C nor C++ allows you to derive pointers pointing outside the original object (as in, memory allocation), it's UB. Yes, the arithmetic operation itself has UB, not merely later derefering an OoB pointer. The only exception is one-past-the-end pointers which are safe to create (but not to dereference of course).

Whether it's safe to "smuggle" pointers around as ints, without doing any arithmetic on them, is "complicated" and honestly there's no one single answer as far as I know.

The other really complicated part is that it is very common to have intermediary ints that have very little relation to the ptr that they come from/turn into (e.g. XOR linked lists are probably legal even though they have ints that mix provenance of 2 completely separate pointers).