You have to subscribe with the Apple TV+ channel specifically for it to play in the Apple TV app (confusing, I know). Not all services offer a TV+ channel, unfortunately.

Hmm... I guess I'll check into that next time and pay more attention. I subscribed to Peacock+ through a bundle offer with Apple TV+, and went through Apple to trigger the purchase. It forced me over to Peacock to make an account there and other than billing it seemed totally separate.