It's (likely) simpler with zoxide + fzf.

I use autojump, which is a lot like zoxide (possible predates it). It stores all the directories you've visited in an SQLite DB (along with the rank for each). I wrote a shell keybinding that presents me with fzf, along with the directories I've visited, in rank order.

With just a few keystrokes, I can visit any directory I've ever visited, really fast. It doesn't need to be the top ranking directory for my query.

I can't live without it now.