pip doesn't do an actual smart solution but it just bruteforces it. The way of pip installing packages is to download the package, execute its setup script that imperatively installs dependencies. There is no proper metadata in PyPI ecosystem. There is some but not enough so pip doesn't know what a package needs until it runs the setup script.

This is the reason Conda exist. It is a proper dependency system with all the dependencies recorded in a package's metadata. So it is possible for the package manager to query and know which dependencies a package needs and what the current environment is and then find a set of packages to install.

This is also why it is surprisingly easy to break your dependencies when installing lots of stuff in pip: sooner or later you're going to end up with some dependency-of-dependency incompatible with a dependency. The practical solution is that you don't install everything in the same environment but set up a separate virtual environment for each project, and then install only what you need there.