My first layout was AZERTY (french one), and I added Colemak 4 years ago. I am slightly faster in AZERTY compared to Colemak still (100wpm/90wpm), I never lost the AZERTY, the key is to just use both all the time. I have a shortcut and I alternate constantly, because I need to write french, or because I wanna code.