It's a mixed bag. Some programs use Green Hills Integrity, some use Wind River VxWorks, some roll their own. I've done all of the above.

The main purpose of the OS is to centralize, schedule, and manage the resources needed for the mission. It's usually pretty lightweight. Different philosophies are used on different missions. The OS risks can be mitigated. Usually there's a backup "golden copy" OS that can boot if needed. There's also "Safe Mode", which prioritizes communications with the ground, so anomalies can be worked.