| ▲ | PyPI's redesigned project pages with clearer security signals are now live(blog.pypi.org) |
| 148 points by rbanffy 4 hours ago | 46 comments |
| |
|
| ▲ | time4tea 2 hours ago | parent | next [-] |
| Not very keen that the whole story of trust is basically centralisation - requiring "big cloud" and essentially promoting lock-in. Did I get that right? Seems a bit anti-pythonic. |
| |
| ▲ | duskdozer 2 hours ago | parent | next [-] | | I think at this point the group of devs+ who are not fully on board with control by and dependence on big tech/VC is a small niche unfortunately. | |
| ▲ | 125ajh 2 hours ago | parent | prev | next [-] | | You got that right. In this thread we have a ClosedAI employee promoting ClosedID services. Look at the sponsors of ClosedID: https://openid.net/foundation/sponsoring-members/ We know why PyPI ripped out PGP support and has refused any verification mechanisms that other distributions use for decades. | | |
| ▲ | cozzyd 2 hours ago | parent | next [-] | | I trust myself way more to with with a pgp key on my computer than figuring out how to use GitHub actions properly... | | |
| ▲ | theteapot an hour ago | parent [-] | | The problem is no one else does. Github workflow is at least public and auditable. Your personal computer is not. |
| |
| ▲ | bbor 44 minutes ago | parent | prev [-] | | 1. "ClosedAI" is not gonna win people over to your side, IMHO. Apple doesn't plant trees, Microsoft makes big hardware, and Python is not satanic. 2. I'm not entirely sure what the link is supposed to prove -- would appreciate a word of clarification if you find the time. It almost feels like a dogwhistle of some kind, but A) I doubt anyone on HN is anywhere near that problematic, and B) there's no clear pattern in those companies other than "interested in ID tech", and none of the usual dogwhistle targets made the list. Also, y'know, OpenAI isn't on it...? I guess my fundamental confusion isn't about the list and more about what "we know". Are you implying that they are against pure, unmanaged PGP for this system because... I mean, why? Because they're playing the long game to promote a verification/attestation/identification (?) technology that you think offers less stringent privacy guarantees, and thus to sell more ads in 2036? If so, doesn't that seem like a kinda roundabout way to raise Display Ads revenue, otherwise known as "that one dying industry"? |
| |
| ▲ | bbor an hour ago | parent | prev [-] | | It's anti lots of things, but I don't think it's anti-Pythonic. Python is about pragmatism & empathy if it's about anything -- bitcoin and such don't factor into it. See: Astral/OpenAI, Anaconda, most of the PSF board historically being employees of big corporate users of python, NVidia's relation to the ecosystem in general, etc. As a passionate anarchosyndicalist, I, for one, am thrilled about it. Decentralization is a tool in our toolbox for identifying and dismantling unjust hierarchies, but it is not a panacea, and can be quite damaging when taken to the extreme. See: cryptocurrency speedrunning 200y of bank regulations in 20y. To put my trust for Python in more concrete terms: they support GitLab. As a passionate GitLab shill, I really couldn't be more impressed! It continues to shock me how infrequently the second most popular forge is acknowledged, much less supported. This shows they're not just, IDK, bowing to Microsoft's demands. | | |
| ▲ | woodruffw 24 minutes ago | parent [-] | | > This shows they're not just, IDK, bowing to Microsoft's demands. Not only don't we, but I must emphasize that Microsoft did not pay for anything involving the implementation of either Trusted Publishing or attestations on PyPI. I wrote a personal blog post explaining some of my own rationale for why we prioritized GitHub and GitLab for PyPI's initial support targets, and it shouldn't be surprising to anybody who knows me that the reasons are pragmatic, not dogmatic[1]. [1]: https://blog.yossarian.net/2024/11/18/Security-means-securin... |
|
|
|
| ▲ | djoldman 3 hours ago | parent | prev | next [-] |
| Some examples: dlt (no provenance): https://pypi.org/project/dlt/#files datasette (yes provenance): https://pypi.org/project/datasette/#files |
| |
| ▲ | MeteorMarc 3 hours ago | parent [-] | | The source distributions have icons while the wheels have not. Strange and confusing. | | |
|
|
| ▲ | samhclark 2 hours ago | parent | prev | next [-] |
| This is great! I want sigstore-based or at least sigstore-style "keyless" short lived signatures to become more and more popular, so that support gets better everywhere. In particular, if having secure OIDC handling in a CI/CD pipeline becomes something that keeps people on GHA or GitLab then I'd hope third-party hosted CI/CD services begin to take that gap seriously and work on meeting those requirements. I'm bought in on keyless signing, and I want there to be many more providers, so I like to see more adoption which builds more momentum. |
| |
| ▲ | ImJasonH an hour ago | parent [-] | | Tangled and Codeberg both seem to support issuing JWTs during actions runs. Codeberg: https://codeberg.org/mfenniak/oidc-jwt
Tangled: https://tangled.org/docs/spindles/workflows/#audience I think OIDC tokens from workflows is basically table stakes at this point, since the alternative is a static secret, which is basically guaranteed to leak eventually. All that's left is for PyPI to accept these tokens in their set of trusted publishers. | | |
| ▲ | woodruffw 21 minutes ago | parent [-] | | Please see the other comments in this thread (as well as on PyPI's issue tracker) for some important context here. It's not enough to throw JWTs over the wall, PyPI also needs to know that it can trust any given service to operationalize the ugly and risk-prone parts of managing PKI. (I haven't looked at Tangled at all. If Tangled has a secret management story for its PKI that's stronger than a private key on disk and can demonstrate that a non-trivial number of PyPI users would benefit from federation, then it's a strong candidate.) |
|
|
|
| ▲ | mhils 3 hours ago | parent | prev | next [-] |
| Very cool to see PyPI put more emphasis towards exposing provenance info. It would be even cooler if trusted publishing would finally support reusable workflows (https://github.com/pypi/warehouse/issues/11096). |
| |
| ▲ | pdonis 3 hours ago | parent | next [-] | | It would be even nicer if there were a way to have an attestation without publishing with a workflow controlled by one of the tech giants. There's no way for me, someone who has had a PyPI account for many years and has used it for all that time to upload packages, to attach my own public key to my PyPI account and then sign uploaded distributions with my private key so the signature can be verified and count as an attestation. | |
| ▲ | tzlahp 2 hours ago | parent | prev [-] | | If only there had been PGP signatures. Oh wait ... | | |
| ▲ | duskdozer 2 hours ago | parent [-] | | Maybe they'll implement PGoogleP, where your private key is safely and securely stored on Google's servers |
|
|
|
| ▲ | cowlevel 4 hours ago | parent | prev | next [-] |
| This is about detecting formerly benign projects that have had their pypi entries hacked, but not their private keys. It does not detect malicious projects. |
| |
| ▲ | Kwpolska 3 hours ago | parent [-] | | And about detecting projects which dared to move their release builds away from GitHub.com/GitLab.com. |
|
|
| ▲ | darkamaul 3 hours ago | parent | prev | next [-] |
| Not directly the subject but they switched from Material for MkDocs to Zensical. It’s quite sad to see that one of the main (or the biggest?) documentation system for the Python completely alienated its user base and led to this fork. |
| |
| ▲ | duskdozer 2 hours ago | parent [-] | | Judging by the readme and homepage, there were probably some large differences brewing. Can't say I'm a fan of how it looks like they'll take the UI |
|
|
| ▲ | eesmith 3 hours ago | parent | prev | next [-] |
| Am I correct in my understanding that the only trusted publishers are for-profit US companies? And these all use proprietary implementations which cannot easily be replicated elsewhere? Like, if I want to host on Codeberg, or Sourcehut, what is the process needed for those platforms to become a trusted publisher? As it happens, I build my releases on my own hardware using a pypa/manylinux release. Can I provide attestation, or am I required to use a third-party service to get that "x Without provenance" removed from my entry? |
| |
| ▲ | woodruffw 3 hours ago | parent | next [-] | | Support for Codeberg is being discussed here[1]. I think there's broad consensus among PyPI's maintainers and admins that more publisher providers are desired, but we also need them to demonstrate strong key management and rotation controls due to the degree of trust and access that becoming a publisher provider grants them (effectively, they become the PyPI credential broker for the projects that use their service). (If you pan through that thread, you'll note that one of open issues with Codeberg is that their underlying server wants to store OIDC private keys on disk at the moment. This is okay for small deployments of personal services, but not for something that PyPI needs to federate with and trust.) [1]: https://github.com/pypi/warehouse/issues/19786 | | |
| ▲ | eesmith 2 hours ago | parent | next [-] | | So I, as someone who does my builds locally, can never have attested builds? Will PyPI be presenting warnings or statements of caution which might cause users to be concerned about using my distributions? Or imply that my build practices are somehow inferior to depending on a third-party build system? | | |
| ▲ | guipsp 2 hours ago | parent | next [-] | | Yes, that is right - that's the purpose of attesting builds. Do you see any other alternative? | | |
| ▲ | woodruffw an hour ago | parent [-] | | The current formats (publish and SLSA provenance) are for attesting publishing and build steps, respectively, but there's nothing stopping PyPI from adding a different, new attestation statement for local builds (something that conveys "I did this locally, and I was this principal (email address) when I did it"). |
| |
| ▲ | woodruffw 2 hours ago | parent | prev [-] | | There's a discussion thread for email attestations here[1]. Those would support local builds. But to take a step back: if you want attested builds because you think attestation is itself a gold star, then you're working from the wrong priors. Attestations do not imply that a package is secure any more than a hash does; what they are is a cryptographic signature over a set of claims, which allows for authenticated auditing of a project's state over time. I think PyPI's documentation does a good job of conveying this nuance[2], and that much of the opprobrium of attestations comes from people either not understanding that nuance or imputing conspiracies into it. [1]: https://github.com/pypi/warehouse/issues/18007 [2]: https://docs.pypi.org/attestations/security-model/#trustwort... |
| |
| ▲ | tzlahp 2 hours ago | parent | prev [-] | | Of course: In principle we'd welcome a change, but there are technical difficulties in perpetuity. | | |
| ▲ | woodruffw 2 hours ago | parent [-] | | Storing your private keys somewhere other than a regular file on disk is not a perpetual technical difficulty. Please assume good faith. |
|
| |
| ▲ | eddythompson80 2 hours ago | parent | prev [-] | | Sounds like it. Currently it seems you'll need to setup your own OIDC provider[0], then work with PyPI to get them to trust your instance as a "Trusted Publisher". It seems like a system to scale for multi-tenant publishers (like millions of GitHub or GitLab or Google build or ActiveState users). The design itself should be able to scale for individual OIDC providers through webfinger and OIDC discovery like Tailscale does it[1], but it doesn't look like it's there yet. Not sure if it's planned either the github issue mentiones future PyPI verified email attestation, though I'm not sure if that implies webfinger/OIDC like ts does it. [0] https://docs.pypi.org/trusted-publishers/ [1] https://tailscale.com/docs/integrations/identity/custom-oidc | | |
| ▲ | woodruffw 2 hours ago | parent [-] | | > The design itself should be able to scale for individual OIDC providers through webfinger and OIDC discovery like Tailscale does it[1], but it doesn't look like it's there yet. This is the right idea, but it's actually not that simple unfortunately: Trusted Publishing relies on per-IdP claims that are more precise than the `sub` claim (because doing pattern matching on the `sub` claim is brittle and prone to security problems). So we require publisher providers (OIDC IdPs) to provide distinct claims that we can match on. For GitHub and GitLab that's things like the repository name, workflow filename, etc., while for others it might be distinct username and immutable ID claims, etc. This is all downstream of OIDC-driven federation being a much weaker and more bespoke process than all parties would like it to be. I think the discourse around this could be much improved by people taking the time to understand these complexities and why they pose a challenge to groups with limited engineering resources (like PyPI). |
|
|
|
| ▲ | s_u_d_o 2 hours ago | parent | prev | next [-] |
| Great news! Would love to see provenance on Homebrew (brew) as well. |
| |
| ▲ | woodruffw 2 hours ago | parent [-] | | Homebrew actually has full provenance. We added it in 2024 (I think, that’s from memory). We never flipped the default verification policy because of some integration concerns, but my understanding is that various downstreams and MDMs do consume and verify Homebrew’s attestations. Edit: I should qualify: Homebrew’s own processes verify its attestations. But that’s not enabled by default on end user machines because it slows installations down and currently requires the `gh` CLI. Neither of those is a tenable long term constraint for a default feature. |
|
|
| ▲ | zzzeek 3 hours ago | parent | prev | next [-] |
| let me tell you something incredibly important about the attestations / trusted publisher thing: if you have any of your files published by a trusted publisher with attestations, then you must have all of your files published that way (in that package). otherwise pypi determines your project may have been compromised and puts some big red warnings up like [1]. it does not try to look at "this particular source file was never part of trusted publishing" or anything like that. and, CRITICALLY, it does not send you an email or anything like that about the non-acceptable file that you just uploaded, which it REALLY should. I mean the model is, they think these uploads are malicious by an attacker. Emailing the project maintainer(s) would be pretty important! (When pypi wanted to get rid of GPG signatures, they started sending me an email for every single signature that got uploaded. so i know you guys know how to send emails). it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that). It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this). [1] https://fosstodon.org/@zzzeek/117554513935692558 |
| |
| ▲ | woodruffw 2 hours ago | parent | next [-] | | > it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that). It's not[1]. > It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this). As mentioned in several other places, we're always interested in adding more publisher providers, provided they meet our key management and notability requirements. (This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.) [1]: https://discuss.python.org/t/pypis-new-security-indicators-a... | | |
| ▲ | zzzeek 2 hours ago | parent | next [-] | | > This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.) I never claimed pypi is trying to "support large for-profit corporations". I fully understand the "large corporations as trusted publisher" thing is because this particular subset, e.g. github, gitlab, google, are widely known institutions with "good" reputations (in this area). The issue is that pypi is not, as of yet, supporting individual projects that want trusted publishing without relying on a large for-profit corporation. I'm not too put off by it but it is definitely a thing that a private for-profit host overnight becomes horrible (see e.g. bitbucket) and I have to get off that company ASAP. Having some non-profit entities here would be extremely helpful. | | |
| ▲ | woodruffw 2 hours ago | parent [-] | | > Having some non-profit entities here would be extremely helpful. I fully agree. I am hopeful that Forgejo or whatever other underlying component of Codeberg can improve its key management primitives here, which would make it very easy to argue for their inclusion as a federated party. |
| |
| ▲ | tzlahp 2 hours ago | parent | prev [-] | | You want the ones that employ core devs through Microsoft or are approved by OpenAI, which codeberg.org is not. |
| |
| ▲ | tzlahp 2 hours ago | parent | prev [-] | | This should be the top comment. PyPI as usual screws up and spreads misinformation about projects. |
|
|
| ▲ | LdBaringSwallow 2 hours ago | parent | prev | next [-] |
| > https://blog.pypi.org/posts/2024-11-14-pypi-now-supports-dig... Thought we already learned years ago an IP address is not a person? That is to say; what the screen on my end communicates is not guaranteed to be representative of ground truth. The IT world was recently making fun of Sam Altman's eye scanner because remote attestation can be faked. All the layers of abstraction wrapped around the code is security through obscurity. The code itself must still be verified as safe. The cloud era has resulted in a lot of people operating at layers of abstraction so far removed from physics this shit has more in common with baseless dogmas, gospels, and spiritualities at this point. We anthropomorphized the cloud inserting ourselves into it. It's just a bunch of networked computers. All this stuff is a spoon that doesn't exist. |
| |
| ▲ | bonoboTP 2 hours ago | parent [-] | | You're speaking in such shorthand that I have no idea how to untangle what you're actually trying to say/propose. I suppose you're saying that attested code can still be malware because malicious actors can also go through the attestation process? | | |
| ▲ | rcxdude 2 hours ago | parent [-] | | Yeah, this check basically just proves that you hosted your source code on a big platform and then built the pypi package with their CI. This is not particularly strong as a trust signal. Especially since it doesn't particularly protect against compromise of access to the code repository. I worry that this is going to distract from better approaches (like reproducable builds and/or distributed review). |
|
|
|
| ▲ | mistursinistur 3 hours ago | parent | prev [-] |
| Nobody is looking at these pages |
| |