...except ruler-mode uses dragging of middle mouse button to move the fill column, which on my system scrolls.

Then simply change the keybindings.

Use `C-h k` and then middle click on the ruler-mode’s header to find out what command is bound to the middle mouse button, then bind that same command to a different mouse button:

    (keymap-set ruler-mode-map "<header-line> <down-mouse-3>" #'ruler-mode-mouse-grab-any-column)