It can't be a guarantee. Even the smallest patches for vulnerabilities change the behavior of the code. Most of the time this is not a problem, but weird things happen all the time. Higher memory usage, slower performance, some regressions that are only relevant for a tiny amount of users, ...

Pretty much. Everything is a breaking change to someone. Best to just ignore sem ver and have a robust automated test suite and deployment process that minimises issues with a bad build.