That's not ESP-IDF, that's the Arduino wrapper around ESP-IDF.

Setting the baud rate does actually do something (just follow the baudRate parameter here: https://github.com/espressif/arduino-esp32/blob/master/cores...), it just doesn't affect the output speed of the USB serial output.

If you specify non-default pins for the serial output (i.e. you're opening a second serial connection to talk to another piece of hardware) the begin() method does influence the data rate.

Also, there are dev boards out there that do actually use a TTL converter and USB-to-serial converter to access the standard serial port, though they're not very common as far as I can tell, so I wouldn't just assume the baud rate setting does nothing on the standard pins either.