Interesting. My experience is almost the same. But I wouldn't call the experience is poor at all. The biggest advantage is that typing feels smoother with Colemak.

The fact that just learning to touch type in any layout is what contributes to the speed is probably right. But then again, my experience is that speed is primarily a function of practice and much less of technique. I remember reading some AMAs by someone who types at 200+WPM and they mentioned that they use QWERTY and they don't touch type.