It's all about reducing the trusted computing base, both in terms of lines of code and of organizations supplying said code.

Most people's computes run a lot of software, including a general-purpose OS with a huge attack surface and third-party applications, often with broad or even unconstrained access to all their files.

General purpose computers often also don't have tamper-proof secure storage; hardware wallets generally do.