Not sure what your exact task is, but I have a similar goal as well. Haven't had time to try alot of different models or ideas yet because got busy with other stuff, but have you tried this: https://youtu.be/dQ-4LASopoM?si=e33FQd5f4fYr4J5L&t=299

where you stitch two images together, one is the working image (the one you want to modify), and the other one is the reference image, you then instruct the model what to do. I'm guessing this approach is as brittle as the other attempts you've tried so far, but I thought this seemed like an interesting approach.