The only time I used -march=native was for a university assignment which was built and evaluated on the same server, and it allowed juicing an extra bit of performance. Using it basically means locking the program to the current CPU only.

However I'm not sure about -O3. I know it can make the binary larger, not sure about other downsides.

-O3 also makes build times longer (sometimes significantly), and occasionally the resulting program is actually slightly slower than -O2.

IME -O3 should only be used if you have benchmarks that show -O3 actually produces a speedup for your specific codebase.