I know someone who has used this trick to get a pay rise whilst not looking for work.

You wait until your boss is in earshot, get someone to ring you and the walk quickly away from your desk saying "yes, yes I'm still interested... Just a sec".

It might make your boss actually consider the reality of replacing you.

I'm sorry that just comes across as unprofessional, weak and passive aggressive. If someone started doing that on my team I'd take it as part of the case against them not a reason to fight to keep them. Also presumably that's in ear shot of other team members, it's disruptive to team morale. If you are serious about looking elsewhere, make it clear you want to stay but xyz is making you consider other options. Do it in private with the right people. Or say nothing at all.