> But I still used the up/down arrow keys for years or scrolled up when I was looking for a previous command

It's not even that bad of a habit with zsh either where it will only scroll through history matching a substring in your history, for instance if I type "rg -i" and then start pressing up arrow, it will only cycle through history entries starting with "rg -i".

if you enable history substring search in zsh you can get that feature anywhere in the command and not just the start. it's very useful if you remember some snippet of the command like it had the word debug in it etc. in fish that is built in and on by default.

though one function that i recently added will narrow your history down based on a space separated set of filters which i found useful

    function hgrep
        set -l cmd history
        for pattern in $argv
            set cmd "$cmd | grep -- "(string escape "$pattern")
        end
        eval $cmd | cat
    end

Clearly I am missing some setting. On zsh on macOS and I tried what you just said and it doesn't work like this