Actually it's not that hard. The main legal avenue to invalidate the Biden pardons would be to argue that Biden never actually issued the pardons. He didn't sign them (the autopen did) and given his mental state there's no clear evidence that he was even aware of them. Trump, in contrast, routinely signs executive orders and pardons with news cameras in the Oval Office with him, while chatting with journalists. So it would be hard to argue that Trump was not the one actually issuing them.

If that's all it takes to invalidate a pardon, it would indeed severely weaken the power.