I don't know if I would go that far without qualifiers. They definitely didn't do the same work load as many modern humans from pretty much all accounts, but that doesn't mean they always did things faster because they didn't need to.

Just for a modern example like painting a room, if im working as a painter as a job, paint is flying off my roller as fast as it can. But if im painting a room for myself, im likely working significantly slower and sedately and not wearing myself out over it. The same for doing other self-sufficient tasks like chopping wood, or washing or mending clothes, maintaining your home and property, or cooking a meal. As a modern job its super fast paced, for someone doing it for themselves without a clock or boss standing over their back they are going to go at a more leisured pace, and likely also enjoying the task far more which could partially count as leisure time. And even if you aren't a farmer and have a boss in those times, if your job was that much harder than a farmer you would likely just leave and find a farm to work on instead.

And of course some tasks are highly seasonal and can't be done at a real leisured pace, certain harvest and planting tasks. Of course those are only for short spurts, and we also have to consider the physical limitations of humans with poorer nutrition who literally could not do the same workload as a modern person. So even the rush at harvest time might be considered a slower pace than many modern jobs. Like a not very healthy by modern standards construction worker today likely has 8 inches height and significantly more muscle mass than the average farmer laborer from 1000 AD, just thanks to the diversity of their diet.