> What if the file name is not valid UTF-8, though?

Then make it valid UTF-8. If you try to solve the long tail of issues in a commonly used function of the library its going to cause a lot of pain. This approach is better. If someone has a weird problem like file names with invalid characters, they can solve it themselves, even publish a package. Why complicate 100% of uses for solving 0.01% of issues?

> Then make it valid UTF-8.

I think you misunderstand. How do you do that for a file that exists on disk that's trying to be read? Rename it for them? They may not like that.