In my case I have almost all notifications disabled so maybe there's an option somewhere. Generally find those notification badges too powerful for me to not check and then get waylaid doom scrolling/watching, so I've made it a habit to always disable them everywhere.

Somewhat tempted to re-enable it as I only really comment on videos that are for very very niche communities and I'm usually answering or asking questions.