Is there a peripheral you can plug into the USB port that exposes gpio pins?

Yes, USB to GPIO adapters are available. Note that you won't be able to implement anything that has timing requirements - you'll need a microcontroller for that.

I think in practice the timing jitter and the low cost difference to something like an Arduino Nano or Pi Pico means people use one of those even if they could technically use a GPIO adapter.

Any dev board like Arduino can work. E.g. by flashing firmware that controls GPIO via Serial.

Look at the FT232H. Does SPI, I²C, UART, JTAG, and GPIO. It’s just a chip but there are hobbyist breakout boards for it.