Having horizontal scroll is annoying, yes, but I don’t think it makes a website unusable on a mobile device.

The way to handle elements wide enough to induce horizontal scrolling is to put them in a <div> (or make CSS for the tag in question) and make CSS with overflow-x: scroll; for just that element. An example from one of my webpages:

  pre {max-width: 90vw; overflow-x: scroll;}

> but I don’t think it makes a website unusable on a mobile device

For you :-) Now imagine you have to use a mobile device but have limited control over your hands, and it's difficult for you to do a vertical scrolling gesture without some horizontal drift.

I've seen this in real life. Why not make things easier for that person when you can? Accessibility is for everyone. (General accessibility rule I like — if it's a persistent minor annoyance for me, it's possibly a dealbreaker for someone with an accessibility requirement I've not thought of)

Your suggestion for constraining horizontal scroll to unavoidably overflowing elements is good, though — I do that too, and I've ended up figuring out how to add affordances so it's clear that scroll is possible (shadows that appear and disappear)