There are trade-offs here:

Give up too early -> users will get annoyed because the task would have been solvable if the model pushed harder.

Give up too late -> collateral damage while completing the task A.K.A. misalignment.

Asking for the user input isn't giving up