This is similar to my experience trying to get Stable Diffusion to denoise a photo for me. (AIUI, under the hood they're trained to turn noise into an image that matches the prompt.) It would either do nothing (with settings turned way down) or take massive creative liberties (such as replacing my friend's face with a cartoon caricature while leaving the rest of the photo looking realistic).

I've had much better luck with models specifically trained for denoising. For denoising, the SCUNet model run via chaiNNer works well for me most of the time. (Occasionally SCUNet likes to leave noise alone in areas that are full of background blur, which I assume has to do with the way the image gets processed as tiles. It would make sense for the model to get confused with a tile that only has background blur, like maybe it assumes that the input image should contain nonzero high-frequency data.)

For your use case, you might want to use something like Real-ESRGAN or another superresolution / image restoration model, but I haven't played much in that space so I can't make concrete recommendations.

if all you want is a denoise plugin, you shouldnt be using a general purpose AI- you should be using a specific tool like DxO PureRAW