So does the secure element in Ledger wallets.

The issue is that the HSM itself has no input or output. Securely interfacing with those is not trivial.