The hardware-based routers have low latency. Fortigate advertises under 5 usec forwarding latency for its routers. Linux kernel forwarding is on the order of 10s of usec. However, under 100 usec of latency is negligible over a WAN link, where you're talking ~5 msec latency even on a fast fiber link. The downside of hardware routing is the lack of flexibility and some performance cliffs. On the consumer grade hardware routers in particular, connection setup is handled by a low-power ARM CPU. You have limits on the number of flows you can accelerate in hardware at a time, etc.

I've got a 10G fiber connection, and I swapped out a Fortigate 100F for a server running VyOS. I had performance problems, because the 10G to 1G transition caused dropped packets at the switch. I was able to solve it by shaping the traffic to the 1G devices to handle queuing in the router, which is something this particular Fortigate can't do. (High end routers have algorithms like WRED designed to get TCP to behave nicely on 10G to 1G drops, but I don't want the noise of a Cisco in my basement.)