Even if you apply Goldfish loss or other things like that, they still understand the gist of the thing they're trained.

That's of course the whole point of things like Goldfish loss.