TFA mentions habituation as a mitigation:

”Habituate. Five or six short cold-water dips over a couple of weeks will cut the cold shock response roughly in half, and the effect lasts for months. This is probably the single most underused safety intervention in open-water swimming.”

I've experienced a similar adaptation when experimenting with cold showers. In that sense it was somewhat of a detriment; the cold became less invigorating but just as unpleasant.