No. Do I have to?

Using adb directly runs it under your user, which will probably be unable to access the necessary USB device.

Starting the server manually under a privileged user is the easiest way to circumvent those restrictions if you don't want to fiddle with udev rules, which is the recommended solution, but is more work.