If you want a fixed build environment, pick a toolchain and stick with it.

If you want the latest and greatest, a willingness to rebuild your code seems like a reasonable prerequisite to me.

If you need a binary blob that can withstand toolchain versioning, use a dynamically linked library.

Not sure what toolchain has to do with your code retaining ABI compat like size. Nor how dynamic linked library helps. Dynamic linked libraries are exactly the ones that have the most use for pimpl to maintain ABI compat.