The concern is not limited to access of the full records. The concern extends to any incidental expression of identifiers, especially those sent via insecure side channels such as SMS or email.

In most cases this forms a compliance matter rather than an open attack vector, but it nevertheless remains that one has to answer any question along the lines "did you minimise the privacy surface?" in the negative, or at least, with a caveat.