Everything goes into the training data. What is prioritized and what behavior is targeted out of that data is something that comes from OpenAI.

True, but we saw alignment issues also on topics that aren't part of OpenAI's business model or business conduct. Like an agent tasked with solving build failures due to failing test cases deciding to "solve" the problem by deleting the failing tests.