The kernel itself used to take just as long.

I could build the kernel on a pentium 3 in about 20 minutes.

When you build your own kernel you don't compile all the random crap like token ring support and 30 filesystems nobody uses.