I know this is more or less expected, but the improvement induced by adding a worker diminishes very rapidly... I guess it's not the cpython/numpy's fault but rather the CPU.

The more fundamental reason is Amdahl's law

https://en.wikipedia.org/wiki/Amdahl%27s_law

Even a tiny bit of serial instruction will limit the speed up