This was the main driver of VGA memory size for a time - if you spent money on 2MB card instead of a 1MB, you could have higher resolution or bit depth.

if you had a big enough framebuffer in your display adapter, though, X11 could display more than your main ram could support - the design, when using "classic way", allowed X server to draw directly on framebuffer memory (just like GDI did)