In my experience, to piddle around is to fool about, while to simply piddle is, indeed, to potty. So based on that, I see where you're coming from.

I absolutely meant piddle about, but I guess that key word got left out.