perf and look at stack traces (or off-cpu events for waits/locks). also, ebpf