As long as it's a linear address space, adding/subtracting a large enough value to a pointer (array, stack) could still cross into another "segment".

but those wouldn't be mapped unless you have crossed a call gate that enabled them. the kernel call gate implementation changes the VM map (region visibility) accordingly