unsafe rust still has to follow invariants, you're just promising the compiler that it does

Yes, clearly it's a matter of using different facilities that may only be accessible to Unsafe Rust, and changing the interface accordingly. But to state that Rust as a whole has such restrictions is not correct.