Data memory-dependent prefetchers like in Apple's newer chips I think only work with full pointers and not offsets, so it could be a decent perf hit.

Still depends. If the indices are pointing to a dense, flat, contiguous array, it will still be faster than following pointers into scattered heap allocations, with or without prefetching, because of how CPU caching works.

Indices can be much smaller than pointers (which are 8 bytes), so they have plenty of cache advantages of their own which can make up for that.