I still would be wary, even in that case. Go does not guarantee that the address of an allocation won't change over the lifetime of the allocation (although current implementations do not make use of this).

If you really store just references to the same arena, better to use an offset from the start of the arena. Then it does not matter whether allocations are moved around.