Each memory controller interface is a not-insignificant number of PCB traces. Increasing the number of memory controllers may dramatically increase the number of PCB layers (or may not, it really depends on the CPU pinout) but it definitely will increase the number of pins on the CPU socket.

This is one of the main reasons (the other is the number of PCIe lanes) why high end desktop and server CPUs have like double the number of pins and so much bigger sockets as compared to consumer desktop CPUs.

Then what's about using 4-8 cheapest motherboards with 64Gb DDR4 and a cheap CPU, and connecting them via PCIE x16 sockets?

And as for DRAM channels, typical cheap motherboard has 2 channels and 4 slots, it should not be super difficult to add 2 more channels.