Meanwhile today, graphics driver tweaks for specific games has become a regular feature and is advertised in the driver release notes (and I guess that's one reason why graphics drivers have become so massively big).

I can't remember where I heard this from but I remember a talk from some guy from the DirectX team and he said that so many games use completely illegal and unsupported/weird API options and calls.

He said that's what 99% of the game optimizations are. Putting stuff in place to fix broken API stuff.