A lot of Ukiyo-e wood prints have small details that mean a lot to locals. I enjoy learning about them on the NHK's English channel.

In this case, the boats are fast (each has a bunch of crewmen) and were used to catch valuable fish. And the boats on the right have two people not at work (barely discernable in TFA's recreation). Those people were on break, getting ready to replace tired oarsmen. That way, the boat could be moving at all times.