Is the idea that once you isolate a function it decompiles it and then iterates changes until either the recompiled asm matches?

That's one way. I'm not certain that's the way you'd project did it, hard to say without looking at the pipeline. But there are N64 "matching decompilation" projects that do it exactly the way you propose.