Yes and it is very unclear to me what are actually the risks...

I am unfamiliar with "coldcard" but can hackers find out which wallet is used for which account? or maybe it can be inferred by analysing the transaction (if any)?

If the hackers do not know which accounts to attack first there is probably no need to panic right now.