Using distcc networked compilation instead of cross-compiling is reasonably fast too and easier to set up if one isn't familiar with either.

the most frustrating part with having to compile a custom kernel is the maintenance burden (packaging/updating etc.), and not the time it takes to build…

I had a similar issue with networking modules for calico (k8s cni) on both rpis and jetson boards…