Even with a single thread you need to ensure that the object you are accessing, still exists and was not freed.

If you're talking about stack objects, that's what lifetimes are for.

If you're talking about heap, you can accomplish that by restricting that the references can't be used to free or invalidate the memory. But you could still allow them to be mutable.