(Without profiling or looking into this at all) I'd guess this has to with thread creation, inter-core communication/latency, and possibly having to merge results or otherwise interleave operations. SMT is another likely candidate.

Regardless, CPUs are really good at single-thread.