Ok, that makes sense. I am assuming it is not common to be arrested for possessing this kind of over the counter remedy?

Not for the ones you buy in Japan, since those are legal.

But, it's not unheard of to get randomly stopped by the police and searched, especially in touristy areas like train stations. Unless you're a Japanese citizen, you have to show ID, and although the searches are optional, most people agree to them.

For customs, usually a few people from each plane are searched.

Anecdotally, if you're a tourist, they're usually looking for medicine that was legal outside of Japan, but illegal within Japan, with small amounts leading to being detained for 23 days (like in this blog post). And if they decide to prosecute you, you'd probably get a suspended sentence (so no prison time), but you'd get deported and a temporary ban from coming back to Japan.