I've had the opposite problem with several of my t-shirts stretching/expanding going from M to something equivalent to XL size and I fail to understand why.

I am not using a dryer, only a washing machine.

Can UV do that?

I see the same. Usually with synthetic fibers.