What I found is that although some tools are good for transcription and the final result is quite accurate, this is not necessarily true for summaries.

Yes, I routinely read meeting summaries spit out by the so-called SOTA tools and they almost always contain major errors. So if you actually need them for anything serious, it would be wise to keep the audio.