But only the assemblies that changed right? Or would it still be all of them?
No, for a full domain reload it reloads everything. This forces all static variables to reset, makes static constructors run again, etc.