"Guilty" seems like the wrong word. Calling something like exercise a "guilty displeasure" just doesn't seem right
It's much more simply stated as: "displeasure"