That would just be abused by people who want to permanently enshrine a bad status quo. They'll file X really shitty, bad faith challenges, and when they all fail, everyone will be permanently stuck with a bad thing.

Imagine if women's suffrage failed 5 times, and hey, guess we'll never get it, 5 times is the limit.