There's a soft failure-mode for bitcoin where due to the alternating difficulty adjustment, you could end up with people only mining every other 2016-block adjustment.

Let's call this cycle A and cycle B.

If A is too hard, miners drop out, cycle B gets easier, miners flood back, cycle A gets harder.

This results in the hard cycle getting longer and the easy cycle getting shorter.

This isn't completely critical as there is I believe a small damping effect, so it isn't completely lethal to bitcoin, but a key thing about bitcoin mining is that whether other people are mining or not doesn't actually affect your own profitiability.

Other people dropping out doesn't actually mean you get more bitcoins per hour/watt, it only affects the next difficulty adjustment as a secondary effect.

The damping effect is that part of your costs are the hardware, space, depreciation etc. leaving that stuff idle costs money - so it makes sense to mine in the less profitable periods too.

I think you're right, it's counterintuitive but less competition means less rewards to share for those who keep mining. Though transaction fees / hour shouldn't decrease, maybe your share of that is bigger.