It certainly gets in the way. The more dependencies, the more work it is to update them, especially when for some reason you're choosing _not_ to automate that process. And the larger the dependencies, the larger the repo.

Would you also try to build all of them on every CI run?

What about the non-source dependencies, check the binaries into git?