> Three to the exponent of five.

I think that's still ambiguous. The base of the exponent is implied, and there are decent arguments or fields in which the assumed base of 10 isn't universal. In your "Three times e's fifth power." version, you needed so specify e. I think for accuracy it's also needed to specify 10 in that "Three to the exponent of five." case.

(Having said that, there are certainly cases where the 10 as the base of the exponent will be clear/unambiguous from the context. )