I actually tried this workflow with some ebay'ed microcassette recorders - even the really compact ones are bulkier (though not necessarily heavier) than a modern cellphone, plus being non-connected you basically have to set aside some time to transcribe and erase (doable, but if you're specifically using it as a memory assist then having to remember this isn't great.) They did do a good job of "one press and they're operating, and you can confidently believe it" (but you have to pick them up in the right orientation for that to work.)

A modern digital voice recorder would have been a better choice for the experiment perhaps? Pen form factor versions exist, e.g. Olympus VP-20, Philips Voicetracer DVT1600.