> Most standard software engineering jobs don’t require that kind of research activity (although it does require some; product development is a creative process)

This seems to describe what good engineers above the senior level do. Certainly everyone with a PhD I work with who rose through the ranks said that being very senior was a lot like being a good researcher - albeit with much more pressure on execution.

I think it totally depends on the job. It's like a process running on a CPU. I've seen software development roles that are "batch processes," where the developer goes into a cave to crank through his tasks uninterrupted, and then emerges in a week to deliver the results. And others that are "interactive, event-driven processes," where there is a lot of back and forth between product owners, UX, and other stakeholders, and lots of iteration and refinement. And then there is a whole spectrum in between! One size doesn't necessarily fit all development styles.