Yes, this sounds like a simpler solution - the major drawback/advantage would be that you'd need to insert the knob in just the right orientation so that the shaft end fits into the potentiometer's slot. Might be a bit fiddly. On the other hand, the poti would keep its state even when the knob is removed, and you wouldn't accidentally change the value when inserting the knob.

Just to nitpick the nitpicker: the knob's shaft needs to be longer than 15 mm, right? "3.5 mm" is the diameter, not the length of a small T(R)S jack.