I guess it depends on what you build, i feel the most complex part of the deal, that makes me think the hardest, is to figure out what to build. Eg understanding the client, and creating a solution that fits between their needs and abilities. The rest is often a technical detail, yes sometimes you need to deep dive to optimize. Anyway if you miss debugging try debugging people;)