Could they just pick the most popular one and make it the canonical standard through a PEP? Because they seem to be doing two things at once here: making a standard and making a new format. You don’t have to do both.

A new format can fix issues that exist in the existing ones, but does that benefit outweigh the benefit of adopting the most popular existing option and having a free and massive heard start towards getting everyone to adopt it and retire the other four?

First, all of this is past tense; the PEP was accepted in March.

But no, there wouldn't have been a coherent metric for "most popular", and anyway the existing designs were independently trying to solve subtly different problems without concern for anyone else. The new format was designed based on getting the relevant parties to talk to each other and figure out what would be necessary for interoperability.