will elements of arraypool still be tracked by GC with overhead?

Depends on the T.

.NET has value types, explicit stack allocation, low level unsafe programming C style, and manual memory management as well.