In TFA, Jef Raskin claims that the story about the mouse is not entirely correct:

JR: No. I designed it to be graphical from the ground up. But the text portions of the interface, which I also cared about, would have been cleaner. People have put together my dislike of the mouse (confusing dislike for a particular input device with dislike for graphic input devices in general; I personally prefer trackballs and tablets) and my careful attention to text handling to a false legend of my wanting a text-based machine. Andy [Hertzfeld, a major developer on the early Mac team], unfortunately, has not generally gone back to the original documents, and he’s interviewed lots of people about the history of the Mac, but not me. His website is, as a result, full of errors.