> An entry fee that is reimbursed if the bug turns out to matter would stop this, real quick.

It would also stop a lot of genuine submissions unfortunately, as some literally can't pay not just won't pay (for both technical or financial reasons), and adds complexity¹. Each project working this way will need to process a bunch of payments and refunds on top of the actual bounty payments, which is not admin free nor potential financially cost free.

I can't think of an easy answer that would work for more than a very short amount of time. As soon as there is money involved and an easy way to use tooling rather than actual effort/understanding to be involved, many will try to game the system ruining it for those genuine participants. Heck, even if the reward is just credit² rather than money, that will happen. Many individual people are honest and useful, people as a whole are a bunch of untrustworthy arseholes who will innocence you and the rest of the world for a penny or just for shits & giggles.

> Assuming the host of the bug bounty program is operating in good faith

This is a significant assumption. One that is it harder to not be paranoid about when you are putting money down.

> they closed it as "works as intended", because they had decided that an optional password was more convenient than a required password

This does not surprise me. My primary bank (FirstDirect, UK) switched the way I authenticate from “between 5 and 9 alphanumeric characters”³ to a 5-digit pin, and all their messages about it assured me (like hell!) that this was “just as secure as before”…⁴

--------

[1] Needing a payment processing option that is compatible with both the reporter and reportee, at the point of submission. At the moment that can be arranged after the bounty is awarded rather than something a project like curl needs to have internationally setup and supported before accepting submissions.

[2] ref: people submitting several simple documentation fixes, one misplaced comma or 'postrophe per pull request, to game some “pull requests accepted” metric somewhere.

[3] which wasn't ideal to start with

[4] I would accept the description “no less secure than before” if they admitted that the previous auth requirements were also lax.