That sounds like bufferbloat[1]. You can usually address that by using a router that supports active queue management, but it's a little esoteric. Newer versions of DOCSIS also specify support for simple active queue management on the modem, and I think this has become a little bit better in recent years. I used to have Comcast/Xfinity service and they didn't do terribly with regard to bufferbloat. They didn't do well, either, but it used to be much worse.

Some of the cable ISPs also have such asymmetric service that you can use most of the upload bandwidth just with ACKs while downloading. They often use ACK suppression to reduce the number ACKs and use the link more efficiently.

[1] https://en.m.wikipedia.org/wiki/Bufferbloat

Comcast is supposedly rolling out mid split to boost upload. Spectrum has already rolled out high split in some markets