I see several comments saying they use fzf instead of zoxide.

You should use both in tandem!

Several years ago, I set up a keybinding that presents me all the directories stored in the zoxide[1] DB in fzf - in rank order. With just a few keystrokes, I get where I need to be, and I'm not presented with all possible directories - just ones I've visited in the past.

This solves the problem of "I want to go to a directory that's not the most ranked one for the string I typed, and zoxide keeps putting me in the other one".

Once you have this flow, there's no going back!

[1] Actually, I use autojump, but it should work with zoxide as well.