| ▲ | A plea not to use uv-build(blogs.gentoo.org) |
| 47 points by Ravencentric 2 hours ago | 33 comments |
| |
|
| ▲ | woodruffw an hour ago | parent | next [-] |
| To copy my response on Lobsters: (I am one of uv’s maintainers.) The nice thing about Python packaging is that it’s standardized: if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards. With that said, I don’t understand why Gentoo needs to rebuild every single release of uv-build. It doesn’t change as much as uv itself, and it’s intentionally compatible across uv versions. Why not set a policy of only updating once weekly, or even less frequently, with exceptions for important breaking changes or bug fixes? (For context: the only reason uv-build releases as often as it does is because it’s easier to keep its internals in lockstep with the rest of uv. But you can do a normal version bound in your Python build backend metadata and uv will happily use whatever version of uv-build is already present.) |
| |
| ▲ | pdonis 22 minutes ago | parent | next [-] | | > The nice thing about Python packaging is that it’s standardized But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library. Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here. > if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards. As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified. > I don’t understand why Gentoo needs to rebuild every single release of uv-build. Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build. > you can do a normal version bound in your Python build backend metadata Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter". | | |
| ▲ | jitl a minute ago | parent | next [-] | | distros patch the build process when they package stuff for the distro all the time. why would they "have to" do anything? i'm not super familiar with gentoo but it seems to me like any other 90s era distro where the distro mediates the relationship between the user and the upstream developer for all packages, by applying patches and potentially doing whatever build they want to. | |
| ▲ | woodruffw 16 minutes ago | parent | prev [-] | | I linked to some of the relevant standards below. At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way. (And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.) |
| |
| ▲ | p_l an hour ago | parent | prev | next [-] | | The problem is packages that use the build backends, and Gentoo is generally not in the business of maintaining that kind of patch. Admittedly rustification like that is a problem not only for gentoo, I have commercially deployed systems where one of the most annoyings elements of updating is presence of rust in python deps... | | |
| ▲ | woodruffw 42 minutes ago | parent | next [-] | | I think that’s actually a different problem: packages that use uv-build for their backend should use a semver range, which enables the kind of “pick a version and use it” strategy I mentioned before. (This is the default when you initialize a project with uv, so I think someone would need to go out of their way to configure their build backend to request a specific patch version of uv-build.) | | |
| ▲ | pdonis 17 minutes ago | parent [-] | | > packages that use uv-build for their backend should use a semver range This wouldn't necessarily solve Gentoo's problem. The only way to be sure to solve Gentoo's problem would be for every Python project that uses uv-build to specify one particular version. Then Gentoo could be sure that once that release of that project was built once, it would never change. In other words, for a distro like Gentoo, the build backend is a dependency, and if it changes, that triggers a change in the build, even if the build backend change would not actually change the final wheel that got built. | | |
| ▲ | woodruffw 11 minutes ago | parent [-] | | That would appear to be the polar opposite of effective: that turns a global constraint problem (everybody needing to have or install a compatible build backend version) into a global coordination problem (everybody needing to have or install an exact version). I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked. It seems much easier to solve the problem of compatibility than the one of getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday. | | |
| ▲ | pdonis 3 minutes ago | parent | next [-] | | > I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked. This doesn't make sense. Gentoo doesn't "pick" any build backend version. They build each Python project exactly the way the project owner tells them to, using whatever build backend is specified in pyproject.toml for that project. That's why it's such a burden for them if build backend versions change so often and projects don't pin a specific version but just say "use the latest"--because then they have to use what the project owner says, and the project owner says, "Update to a new build backend every week because that's how often my chosen backend, uv-build, likes to push updates". | |
| ▲ | pdonis 6 minutes ago | parent | prev [-] | | > a global coordination problem (everybody needing to have or install an exact version). This coordination problem is already solved by PyPI. That's where Gentoo looks to find the build backend--just like everyone else that builds packages. The project owner had to be able to find whatever version they used there or they couldn't even build their project locally. So they should be able to depend on the fact that anyone else trying to build their project can also find that same version at the same place. > getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday. You're missing my point. If I know that my project builds properly with build backend X version Y, and I pin that version in my pyproject.toml, I don't care how many newer versions you want to release. Unless I make a change to my project that merits updating my build backend to a newer version, I'm going to simply leave my project alone. I already know it will build fine with the version I have. Why should I push an update every Tuesday just because you decided to release a new version of the build backend, which I don't need anyway? |
|
|
| |
| ▲ | arto 40 minutes ago | parent | prev [-] | | I hear you, but realistically the rustification of Python package internals is only going to strongly increase over the next year (I'm working on a number of such packages, myself), so systematic solutions to such annoyances are going to become increasingly necessary |
| |
| ▲ | ForHackernews an hour ago | parent | prev | next [-] | | Is Gentoo one of those compile-everything-from-source distros? | | |
| ▲ | BeetleB 6 minutes ago | parent | next [-] | | With apologies to Slackware, it is the compile-everything-from-source distro. Happy user for over 23 years now! | |
| ▲ | tfrancisl 14 minutes ago | parent | prev | next [-] | | Yes, and unlike NixOS they dont calculate build hashes for the sake of making binary caches. So truly, you have to build from source, even if your recipe is exactly the same as everyone else's. I've never really understood this zealotry around compiling things myself, and boy are there some big (slow to compile) programs out there these days. | |
| ▲ | pjmlp 33 minutes ago | parent | prev | next [-] | | They basically started it all, that was what made Gentoo different from everyone else. | |
| ▲ | r_lee an hour ago | parent | prev | next [-] | | I'm pretty sure that is the entire purpose of Gentoo | |
| ▲ | collinfunk an hour ago | parent | prev [-] | | Yeah, it has binary packages nowadays as well, though. |
| |
| ▲ | crabbone 27 minutes ago | parent | prev | next [-] | | You pretty much opened with a... I'm sorry for a strong word, but it is what it is... a lie. No. Python packaging isn't standardized. Tools aren't interchangeable. And they are all bad, uv including. I wish it didn't exist, so that there were fewer problems for people like me, whose job is, essentially, working around the problems created by people who build tools like uv and people who mindlessly adopt them. To answer to your specific arguments: nobody is going to believe that your proposed solution is going to hold water longer than it takes for milk to go sour. Too many people, too often had burned on Python packaging tools to trust some tool not to break, to be consistent etc. Without even realizing it, you suggested that someone else should use a crutch to work around the problem you created, and you expected them to just go along with the flow and live by the bottom of the barrel lowest possible "standards" that exist in Python packaging world. No wonder they don't want uv-build... why would any sensible person want it? | | |
| ▲ | woodruffw 22 minutes ago | parent [-] | | Python build backends are standardized in PEP 517, and packaging metadata is standardized in PEP 621. You can read the actual metadata PEPs for “computed” metadata i.e. the metadata that appears in distributions as well, if you’d like. (It’s not kind or charitable to call someone a liar.) | | |
| ▲ | 231ag 4 minutes ago | parent [-] | | And every backend accepts a different syntax in the .toml file. You cannot just swap out poetry for flit. Especially when C extensions are involved, everything is entirely unportable. Of course, yet again crabbone is downvoted for speaking the truth. |
|
| |
| ▲ | 231ag 3 minutes ago | parent | prev [-] | | The chutzpah of claiming portability is astonishing. Everyone knows it is false. Does the OpenAI voting ring help here? |
|
|
| ▲ | afdbcreid 3 minutes ago | parent | prev | next [-] |
| By a search, Gentoo accounts for less than 1% of all developers. If uv-build is more conformant to people, probably volunteers (and if they're using it, it probably is), I don't think the Gentoo people have any right to want them not to use it because it makes their lives harder. They can request, and they do request nicely, but they also present the drawbacks for them as if they are absolute, and even claim that "its benefits don’t outweigh the drawbacks". |
|
| ▲ | nimish 35 minutes ago | parent | prev | next [-] |
| I'm not sure that skipping SPARC is a bad thing. Who even has SPARC machines sitting around to validate builds? Even Fujitsu moved to ARM for their later cores. |
| |
|
| ▲ | shadowgovt 7 minutes ago | parent | prev | next [-] |
| I don't think I agree with the author's assessment here: > The other reason is performance and that argument makes even less sense. Even if it mattered, we’re realistically talking about a difference in the ballpark of 100 ms, for an action executed rarely; it is equally possible that any gain will be outweighed by the time needed to fetch the 1.5 MiB wheel for uv-build. Modern CI architectures are building Docker stacks from scratch and using a readthrough cache of wheels. Downloading may still be the dominant factor, but the second-dominant factor does start to add up, especially when the downloading gets cached at various layers. |
|
| ▲ | pjmlp 32 minutes ago | parent | prev | next [-] |
| Wait until he discovers that CPython is actually taking Rust in, see recent PyCon talks on the matter. |
| |
| ▲ | woodruffw 28 minutes ago | parent [-] | | I think mgorny is aware of that; he’s been active on the relevant DPO threads. (At the end of the day, this is exactly what support tiers are for. CPython, at least to my eyes, has been very transparent about not guaranteeing support for every platform that has a C compiler.) |
|
|
| ▲ | throwawayy6767 an hour ago | parent | prev [-] |
| Maybe it's time to grow up and download binary builds like a normal person. I had my 'let's compile the linux kernel from source for 3.5% hypothetical performance gains' phase but you can't expect professional devs to cater to source-obsessed hobbyists |
| |
| ▲ | woodruffw 6 minutes ago | parent | next [-] | | Gentoo’s decision to support source builds seems eminently reasonable to me, and I don’t think we should reward it with condescension. The bigger problem, to me, is that it’s not really clear why this is actually a problem for Gentoo. I am glad they perform user-facing source builds, however, because they see things that other distributors don’t. | |
| ▲ | bmenrigh an hour ago | parent | prev | next [-] | | I doubt this oft-repeated stereotype was every particularly accurate. I'm not aware of any current Gentoo users on the distro specifically for performance reasons. Gentoo users are in it for control over their system and customization. | |
| ▲ | Cyph0n 34 minutes ago | parent | prev | next [-] | | Or adopt the Nix approach of caching common binary artifacts in a content addressable store and rebuild from source only when needed. | |
| ▲ | lostapathy 44 minutes ago | parent | prev [-] | | Gentoo made a lot of sense WAY back in a pre-docker world, where you had to run the whole world on the host OS. Being able to compile it all meant you could link it all together with whatever weird combination of optional libraries you wanted. Things like getting sendmail to auth over LDAP, that nobody packaged in the base distro, become possible. And with Gentoo's tooling, you could build a coherent distro to your own unique demands. Now though? It's way easier to have a herd of unicorn docker containers running that each handle one weird thing in isolation. | | |
| ▲ | weinzierl 33 minutes ago | parent [-] | | The herd of docker containers would in practice share almost nothing apart from the kernel and waste a lot of RAM. Didn't matter when RAM was cheap, but nowadays... |
|
|