Is jpeg the only format which can be “rotated losslessly” in this way, ie without exif metadata (and excluding any obvious formats like bitmaps or whatever can be trivially rotated) ?

From what I know it can't be "rotated losslessly" in all cases, only if the dimensions of the images are multiples of the MCU which are block of pixels whose size is determined by the chroma subsampling. Ex.: With the common "4:2:0" subsampling the MCU is 16x16 the image's height and width must be exactly multiples of 16, otherwise I think it's just visually lossless and uses some tricks that I'm still not sure how they work.

Both PNG and Webp exist in lossless formats which can be easily rotated without loss of quality. Jpeg is different, because its internal structure consists of 8x8 blocks (which, BTW, can be rotated lossless if both sides of the image are multiples of 8, iirc.

For photos all these assumptions are rather theoretical IMO, because modern cameras create images in sizes which almost always have to be reduced in size for viewing, unless one wants to pixel-peep.