It's the reverse.

As price per coin goes up, more folks will find mining profitable and invest in mining operations. Difficulty goes up until it's no longer attractive for anyone to add to the global hash rate.

As price per coin goes down, less of those operations are profitable and fewer new people will find it to be a good investment. Difficulty stays the same or goes down. Due to capital expenses, difficulty is more sticky in the downward direction than upwards.

There is of course some marginal price action in between where there is in theory selling pressure from miners when it's less profitable to mine (to fund operational expenses and debt), but I don't think it's super material to the overall market volume these days.