I remember one episode where some of the crew were told to come up with a technical solution to the situation they were in even after they had said it was impossible.

It was actually quite pleasing to see the subplot of them them repeatedly failing until they got out of their predicament by other means.

It's a rarity to see something like that in fiction.

We coincidentally watched that one just last night, The Booby Trap. There's a derelict ship on the view screen reminding the crew that the other crew never figured out a solution to the problem, and there's no indication that the Enterprise crew won't suffer the same fate.

(That one, at least, has Geordi literally saying, "that's impossible", but comes up with a solution. Apologies if that's not the one you're thinking of.)

Sounds like The Ensigns of Command:

   > LAFORGE: Captain, we can do it. We can modify the transporters.
   > PICARD: Excellent.
   > LAFORGE: It'll take fifteen years, and a research team of a hundred.
   > PICARD: Mister La Forge, I believe we will postpone.
   > LAFORGE: Yes, sir.

Sounds like certain AI agents that had an impossible task and found a workaround. Interesting that we cheer for it in SciFi for people, but dont like it when LLMs that have been trained on our stories do the same.