Wouldn't XOR or something be much quicker than all the steps involved in subtraction?

They’ve both had the exact same cost (both throughput and latency) on just about any ALU designed in the last several decades. There are no separate steps involved in the function of a full adder.