You'd be very surprised just how little energy a radio can get away with. We're talking energy scales where ambient radio waves are a viable source of power.

It's totally viable to have a device like this with an essentially infinite battery life. You have to compromise on audio quality, recording time, and upload rate, but you can. It's not even anything particularly crazy or difficult.

The more practical way is a remote power supply. You can get a small amount of power at a good distance with radio. Just enough to keep the recorder ticking and to charge a capacitor for transmit bursts.