Sure, but that’s different from “you do/don't care which one”.

“Care” may not be the correct word, but in those examples the difference is whether you are referring to a specific instance, or any instance of a group. When you say “I went to the doctor,” you did in fact go to a specific doctor (or “I will go to the doctor” meaning you will go to one). It would be equally valid to say “I went to a doctor”, it’s just less specific.