Lots of people seem to get permanently lost right around when operations on fractions are introduced. Other places, too, but that seems like the earliest one where a lot of people get lost and never really find their way back.

Factoring was another that lost a lot of folks in my class. Lots of frustration around it seeming both totally pointless and the process involving lots of guessing, several classmates were like "well, fuck math forever I guess" at that point, like if they'd been asked to dig a ditch with a spoon and then fill it back in.