It does have an impact. Discord blogged about unavoidable GC delays in go that occurred because go needed to traverse live memory, their only option (in go) for reducing that pause was to reduce the size of their data set.