Interesting. My experience is almost the same. But I wouldn't call the experience is poor at all. The biggest advantage is that typing feels smoother with Colemak.

The fact that just learning to touch type in any layout is what contributes to the speed is probably right. But then again, my experience is that speed is primarily a function of practice and much less of technique. I remember reading some AMAs by someone who types at 200+WPM and they mentioned that they use QWERTY and they don't touch type.

You have read my comment inattentively, I consider QWERTY-inspired ones exactly as weak as QWERTY.

Speed is not important in typing goals at all. What maters is ability to delegate a typing routine out of your consciousness. No peeking, no misusing pinkies, no caring about WPM, no mismatching keys from different layouts. You should always prefer to put all of your 10 fingers on the keyboard even if all you need is to type one letter, even being interrupted from sleep, because you should understand that touchtyping is faster than hunt-and-peck even in one-button case.

You either can input your password using touchtyping or not. If you have achieved touchtyping on any layout, no switching helps you to decrease cognitive hardness. Touchtyping should be done in youth, so if you are not cosplaying some idiots you should devote your brain cells to the proper layout at once.

I switched to colemak, but I paired this switch with split keyboards. So qwerty is still easy to use on regular keyboards, but I'm sure I'd get super tripped up if I used qwerty on a split kb.