It's not just the size of the data but also the amount of copies. Consider a huge tree structure: even if each node is small, doing individual "malloc-style" allocations for millions of nodes would cause a huge performance hit.

Obviously for your use case it's not a problem but other use cases are a different story. Games in particular are very sensitive to performance spikes. Even a naive tracing GC would do better than hitting such an implicit copy every few frames.