Have you seen GhostCell[1]? Seems like this could be a solution to your problem.

[1]: https://plv.mpi-sws.org/rustbelt/ghostcell/

The qcell crate is perhaps the most popular implementation of GhostCell-like patterns. But the ergonomics is a bit of a challenge still.

Yes. There's an implementation at https://github.com/matthieu-m/ghost-cell

Not clear why it never caught on.

There have been many attempts to solve the Rust back reference problem, but nothing has become popular.