I find that other people get annoyed or at least bring attention to it when I try to use 24-hour time with them, so that's a downside I guess.

While that is true that is less an issue with 24-hour time and more with humans' tendency for the Familiarity Bias.

I guarantee that if most people grew up using 24-hour time, they would find it just as weird to hear someone use 12-hour.