Try GSK_RENDERER=cairo and see if it helps. I think the memory usage increase will mostly be GPU buffers. If GPU rendering isn't necessary for your app then it's pure bloat to use it, indeed. But with very large hi res screens it's often the case now that CPU rendering can't keep up anymore.

3 MB RSS less with GTK4 4.22.4 on Debian Forky (testing).