It's entirely possible a game manages paging VRAM badly. But allowing the game more flexibility isn't a problem with a larger BAR, it's that the game is stupid and gets dumber the more VRAM you give it.

If a program runs slower when you give it more RAM, the problem isn't giving more RAM.