So both this and litellm went straight to PyPI without going to GitHub first.

Is there any way to setup PyPI to only publish packages that come from a certain pattern of tag that exists in GH? Would such a measure help at all here?

Yes: if you use a Trusted Publisher with PyPI, you can constrain it to an environment. Then, on GitHub, you can configure that environment with a tag or branch protection rule that only allows the environment to be activated if the ref matches. You can also configure required approvers on the environment, to prevent anyone except your account (and potentially other maintainers you’d like) from activating the environment.

Don't have the token on your hands. Use OICD ideally, or make sure to setup carefully as a repository secret. Ensure the workflow runs in a well permission read, minimal dependency environment. The issue with OICD is that it does not work with nested workflows because github does not propagate the claims.