It's quite possible to write a program that uses 32-bit pointers in 64-bit mode, just keep all code and data at addresses below 4G. Such a program will run on any standard x86-64 kernel, because it doesn't use the x32 ABI. x32 is "only" required to support the C library, which expects pointers passed from/to the kernel to be the same size as those in userland.

(Things THEY don't want you to know: you can in fact write code in languages which aren't C, don't compile down to C, and don't depend on a C library. Even under Linux.)

As for reloading segment registers, 64-bit Linux is able to run 32-bit binaries, so there have to be ring 3 code segments for both modes. And there is nothing in the architecture stopping assembly code from jumping between those segments!

With a 32-bit binary that does this, you get access to all the features of 64-bit mode, with everything in your address space guaranteed to be mapped at an address below 4G. The only point where you need to use 64-bit pointers is in structures passed to syscalls. (for arguments in registers it's done automatically by zero-extension)