Why does the number of drivers compiled in affect memory use? Shouldn't the kernel just load what it needs and ignore the rest?

If it's compiled into the kernel it'll occupy memory regardless if it's being used. If it's built as a kernel module then it doesn't occupy memory until loaded.