I think you are conflating two concepts. Most states do not allow recall of state legislators as you mentioned, but recalling federal legislators is not allowed in any state—there is no constitutional mechanism enabling it.

You are quite correct. Reinforcing the notion that the only way to get these people to leave off-season is "voluntarily."