| ▲ | woodruffw 2 hours ago |
| To copy my response on Lobsters: (I am one of uv’s maintainers.) The nice thing about Python packaging is that it’s standardized: if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards. With that said, I don’t understand why Gentoo needs to rebuild every single release of uv-build. It doesn’t change as much as uv itself, and it’s intentionally compatible across uv versions. Why not set a policy of only updating once weekly, or even less frequently, with exceptions for important breaking changes or bug fixes? (For context: the only reason uv-build releases as often as it does is because it’s easier to keep its internals in lockstep with the rest of uv. But you can do a normal version bound in your Python build backend metadata and uv will happily use whatever version of uv-build is already present.) |
|
| ▲ | pdonis an hour ago | parent | next [-] |
| > The nice thing about Python packaging is that it’s standardized But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library. Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here. > if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards. As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified. > I don’t understand why Gentoo needs to rebuild every single release of uv-build. Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build. > you can do a normal version bound in your Python build backend metadata Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter". |
| |
| ▲ | jitl an hour ago | parent | next [-] | | distros patch the build process when they package stuff for the distro all the time. why would they "have to" do anything? i'm not super familiar with gentoo but it seems to me like any other 90s era distro where the distro mediates the relationship between the user and the upstream developer for all packages, by applying patches and potentially doing whatever build they want to. | | |
| ▲ | pdonis 43 minutes ago | parent [-] | | Gentoo builds every package from source on every user's machine, and when each user does an update, every package that changed gets rebuilt from source again. So every Gentoo user that has a Python project that uses uv-build ends up having to rebuild uv-build, and then every Python project they have that uses it, every time uv-build pushes an update. Other distros don't necessarily have quite the same problem because, if they include packages of Python projects, they don't normally track the build backend as a dependency. So if uv-build pushes an update, the distro would push an updated package for uv-build, but they wouldn't look at every Python project they package to see if it used uv-build as a build backend and rebuild it if it did. It's the "rebuild from source every time" aspect of Gentoo that forces that connection. | | |
| ▲ | jitl 30 minutes ago | parent | next [-] | | i guess gentoo is rolling-release, vs something like debian where every package is frozen in time (+ backports), and debian does seem to largely solve the coordination problem by making everything on the system use a fixed version of some package; with rare situations allowing >1 version of the package at a time. my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something. for example, they seem to be fine doing a quick sed on pyproject.toml (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/pa...) or patching stuff for compat (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/as...) This is what i mean when it seems typical for a distro to monkey around with packages and/or their build system. These were just the first two packages i clicked on in dev-python. maybe others are "pure upstream", but there's definitely the capability here to just... not have the problem? | |
| ▲ | BigTTYGothGF 7 minutes ago | parent | prev | next [-] | | Seems like a design flaw in gentoo that previously hadn't been an issue because build tools weren't being updated nearly as often as uv-build is. | |
| ▲ | swiftcoder 30 minutes ago | parent | prev [-] | | > they don't normally track the build backend as a dependency... It's the "rebuild from source every time" aspect of Gentoo that forces that connection I don't follow. This feels like Gentoo is mishandling build-tool dependencies. Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump? | | |
| ▲ | fuzzy2 16 minutes ago | parent | next [-] | | IIRC, rebuilds to GCC outputs are/were only recommended when the compiler "profile" changes. So basically whenever the ABI could no longer be guaranteed. | |
| ▲ | ForHackernews 27 minutes ago | parent | prev [-] | | Strictly speaking, they should, right? Differences in GCC could produce differences in anything it compiles. |
|
|
| |
| ▲ | woodruffw an hour ago | parent | prev [-] | | I linked to some of the relevant standards below. At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way. (And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.) | | |
| ▲ | pdonis 42 minutes ago | parent [-] | | > it seems extremely unlikely that they actually need to build every single patch version of uv. If every new patch version is still within the semver range specified in a project that uses uv-build, then yes, every Gentoo user will have to rebuild that new patch version of uv-build in order to rebuild that project. And since the build backend is a dependency of the project, pushing a new patch version will trigger a rebuild of the project. |
|
|
|
| ▲ | p_l 2 hours ago | parent | prev | next [-] |
| The problem is packages that use the build backends, and Gentoo is generally not in the business of maintaining that kind of patch. Admittedly rustification like that is a problem not only for gentoo, I have commercially deployed systems where one of the most annoyings elements of updating is presence of rust in python deps... |
| |
| ▲ | woodruffw an hour ago | parent | next [-] | | I think that’s actually a different problem: packages that use uv-build for their backend should use a semver range, which enables the kind of “pick a version and use it” strategy I mentioned before. (This is the default when you initialize a project with uv, so I think someone would need to go out of their way to configure their build backend to request a specific patch version of uv-build.) | | |
| ▲ | pdonis an hour ago | parent [-] | | > packages that use uv-build for their backend should use a semver range This wouldn't necessarily solve Gentoo's problem. The only way to be sure to solve Gentoo's problem would be for every Python project that uses uv-build to specify one particular version. Then Gentoo could be sure that once that release of that project was built once, it would never change. In other words, for a distro like Gentoo, the build backend is a dependency, and if it changes, that triggers a change in the build, even if the build backend change would not actually change the final wheel that got built. | | |
| ▲ | woodruffw an hour ago | parent [-] | | That would appear to be the polar opposite of effective: that turns a global constraint problem (everybody needing to have or install a compatible build backend version) into a global coordination problem (everybody needing to have or install an exact version). I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked. It seems much easier to solve the problem of compatibility than the one of getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday. | | |
| ▲ | pdonis an hour ago | parent | next [-] | | > I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked. This doesn't make sense. Gentoo doesn't "pick" any build backend version. They build each Python project exactly the way the project owner tells them to, using whatever build backend is specified in pyproject.toml for that project. That's why it's such a burden for them (or more precisely their users, since each Gentoo user is having to do this) if build backend versions change so often and projects don't pin a specific version but just say "use the latest"--because then they have to use what the project owner says, and the project owner says, "Update to a new build backend every week because that's how often my chosen backend, uv-build, likes to push updates". | | |
| ▲ | woodruffw 42 minutes ago | parent [-] | | > if build backend versions change so often and projects don't pin a specific version but just say "use the latest" This isn’t what’s happening: uv defaults to semver bounds, so a maintainer would have to go out of their way to introduce a troublesome backend constraint. And note: there is no “use the latest” constraint in Python packaging. If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has. (I also know from first-hand experience that Gentoo and other distributions do not build Python packages as-is; they patch them in all kinds of ways. But that’s neither here nor there.) | | |
| ▲ | pdonis 37 minutes ago | parent [-] | | > uv defaults to semver bounds But a new patch release will be within the same semver bound, so it amounts to the same thing. > If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has. What do build frontends actually do, though? | | |
| ▲ | woodruffw 29 minutes ago | parent [-] | | > But a new patch release will be within the same semver bound, so it amounts to the same thing. Sorry, can you lay out the actual problem you’re envisioning for me? If uv-build goes from 0.13.1 to 0.13.2, then anybody with a semver bound of `uv-build>=0.12,<0.14` doesn’t need to update anything. They can keep using whatever version of the build backend they already have that satisfies that bound. The main thing that would be troublesome is if a user pins their build backend to a specific range or version, e.g. `uv-build==0.13.1`. Then, upon bumping it, everyone who has already built 0.13.1 will need to invalidate and rebuild their build backend. But that’s why that isn’t the default. > What do build frontends actually do, though? uv will definitely use a compatible version. It won’t even download a copy of that that compatible version by default, since the uv binary itself is typically compatible and can invoke the build backend directly. I’m not sure what pip and others do, but I would expect they do the same. It’s the obvious way to implement backend resolution. |
|
|
| |
| ▲ | pdonis an hour ago | parent | prev [-] | | > a global coordination problem (everybody needing to have or install an exact version). This coordination problem is already solved by PyPI. That's where Gentoo looks to find the build backend--just like everyone else that builds packages. The project owner had to be able to find whatever version they used there or they couldn't even build their project locally. So they should be able to depend on the fact that anyone else trying to build their project can also find that same version at the same place. > getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday. You're missing my point. If I know that my project builds properly with build backend X version Y, and I pin that version in my pyproject.toml, I don't care how many newer versions you want to release. Unless I make a change to my project that merits updating my build backend to a newer version, I'm going to simply leave my project alone. I already know it will build fine with the version I have. Why should I push an update every Tuesday just because you decided to release a new version of the build backend, which I don't need anyway? | | |
|
|
| |
| ▲ | arto an hour ago | parent | prev [-] | | I hear you, but realistically the rustification of Python package internals is only going to strongly increase over the next year (I'm working on a number of such packages, myself), so systematic solutions to such annoyances are going to become increasingly necessary | | |
| ▲ | p_l 9 minutes ago | parent [-] | | Well, it did force me to work on better build caching and the like, but at the same time, it accelerated my wish to remove python where possible |
|
|
|
| ▲ | BigTTYGothGF 9 minutes ago | parent | prev | next [-] |
| > The nice thing about Python packaging is that it’s standardized Has anybody done the analysis to determine if this is something that can, in practice, be relied on? |
| |
| ▲ | p_l 7 minutes ago | parent [-] | | My somewhat ignorant experience is full of dealing with combinatorial expansion of python build backends and needing to husband them in nixos including all sorts of overrides to deal with, iirc, weird circularities in deps. |
|
|
| ▲ | ForHackernews 2 hours ago | parent | prev | next [-] |
| Is Gentoo one of those compile-everything-from-source distros? |
| |
| ▲ | BeetleB an hour ago | parent | next [-] | | With apologies to Slackware, it is the compile-everything-from-source distro. Happy user for over 23 years now! | |
| ▲ | tfrancisl an hour ago | parent | prev | next [-] | | Yes, and unlike NixOS they dont calculate build hashes for the sake of making binary caches. So truly, you have to build from source, even if your recipe is exactly the same as everyone else's. I've never really understood this zealotry around compiling things myself, and boy are there some big (slow to compile) programs out there these days. | | |
| ▲ | t43562 39 minutes ago | parent | next [-] | | For one thing you can, in theory, optimise for the CPU you have instead of some generic target that has to run acceptably everywhere. | | |
| ▲ | tfrancisl 29 minutes ago | parent [-] | | Sure, but again, you could in theory distribute said builds and stop wasting compute compiling yourself. I know nixpkgs, for example, builds with very generic targets and not super aggressive compiler flags, but at least the machinery exists to do this. | | |
| ▲ | t43562 11 minutes ago | parent [-] | | On my 10 computers for example? :-) Gentoo wasn't aimed at people with such luxuries. I'm not a Gentoo user BTW but I do think building from source is attractive for the same reasons cargo (Isn't that build from source mostly?) and go do it. |
|
| |
| ▲ | NoidFonsense 39 minutes ago | parent | prev | next [-] | | There are binary packages in gentoo, but you can't take advantage of USE flags. | |
| ▲ | yolp5 39 minutes ago | parent | prev [-] | | [dead] |
| |
| ▲ | pjmlp an hour ago | parent | prev | next [-] | | They basically started it all, that was what made Gentoo different from everyone else. | | |
| ▲ | p_l 6 minutes ago | parent [-] | | It also made it nice base for both ChromeOS and its descendants like COS and Flatpak (continuation of original CoreOS) |
| |
| ▲ | r_lee 2 hours ago | parent | prev | next [-] | | I'm pretty sure that is the entire purpose of Gentoo | |
| ▲ | collinfunk 2 hours ago | parent | prev | next [-] | | Yeah, it has binary packages nowadays as well, though. | |
| ▲ | an hour ago | parent | prev [-] | | [deleted] |
|
|
| ▲ | crabbone an hour ago | parent | prev | next [-] |
| [flagged] |
| |
| ▲ | woodruffw an hour ago | parent [-] | | Python build backends are standardized in PEP 517, and packaging metadata is standardized in PEP 621. You can read the actual metadata PEPs for “computed” metadata i.e. the metadata that appears in distributions as well, if you’d like. (It’s not kind or charitable to call someone a liar.) | | |
| ▲ | 231ag an hour ago | parent [-] | | And every backend accepts a different syntax in the .toml file. You cannot just swap out poetry for flit. Especially when C extensions are involved, everything is entirely unportable. Of course, yet again crabbone is downvoted for speaking the truth. | | |
| ▲ | woodruffw 38 minutes ago | parent | next [-] | | Each build backend can have its own extensions to the standard packaging metadata. In practice this is common because Python packaging wasn’t designed to solve language ecosystem problems, but to make the union of maintainers, Linux distros, labs, and everybody else who uses Python somewhat happy and able to preserve existing flows. This is bad for obvious reasons. But it’s not the same thing as Python packaging not being standardized; I would say that Python packaging is as standardized as its user base is willing to stomach. That willingness has grown over the years, to the immense credit of a large group of users who otherwise don’t agree on much. | |
| ▲ | BigTTYGothGF 16 minutes ago | parent | prev [-] | | > Of course, yet again crabbone is downvoted for speaking the truth It wasn't for what was said, but for how it was said, which is why you can say all sorts of heinous stuff on here as long as you're couth about it. |
|
|
|
|
| ▲ | 231ag an hour ago | parent | prev [-] |
| [flagged] |
