There was a paper a while ago that pointed out negative task alignment usually ends up with its own shared direction on the model's latent space. So it's actually totally unsurprising.

Do you recall which paper it was? I would be interested in reading it.