| ▲ | pizlonator 7 hours ago |
| Here's the chasm I want to see Rust cross: Dynamic linking with a safe ABI, where if you change and recompile one library then the outcome has to obey some definition of safety, and ABI stability is about as good as C or Objective-C or Swift. Until that happens, it'll be hard to adopt Rust in a lot of C/C++ strongholds where C's ABI and dynamic linking are the thing that enables the software to get huge. |
|
| ▲ | elteto 6 hours ago | parent | next [-] |
| > Until that happens, it'll be hard to adopt Rust in a lot of C/C++ strongholds where C's ABI and dynamic linking are the thing that enables the software to get huge. Wait, Rust can already communicate using the C ABI. In fact, it offers exactly the same capabilities as C++ in this regard (dynamic linking). |
| |
| ▲ | pizlonator 6 hours ago | parent [-] | | That's an unsafe ABI. | | |
| ▲ | elteto 6 hours ago | parent | next [-] | | As unsafe as C or C++. In fact, safer, because only the ABI surface is unsafe, the rust code behind it can be as safe or unsafe as you want it to be. I was addressing this portion of your comment: "C's ABI and dynamic linking are the thing that enables the software to get huge". If the C ABI is what enables software to get huge then Rust is already there. There is a second claim in your comment about a "safe ABI", but that is something that neither C or C++ offers right now. | | |
| ▲ | pizlonator 6 hours ago | parent [-] | | Here's the problem. If you told me that you rebuilt the Linux userland with Rust but you used C ABI at all of the boundaries, then I would be pretty convinced that you did not create a meaningful improvement to security because of how many dynamic linking boundaries there are. So many of the libraries involved are small, and big or small they expose ABIs that involve pointers to buffers and manual memory management. > There is a second claim in your comment about a "safe ABI", but that is something that neither C or C++ offers right now. Of course C and C++ are no safer in this regard. (Well, with Fil-C they are safer, but like whatever.) But that misses the point, which is that: - It would be a big deal if Rust did have a safe dynamic linking ABI. Someone should do it. That's the main point I'm making. I don't think deflecting by saying "but C is no safer" is super interesting. - So long as this problem isn't fixed, the upside of using Rust to replace a lot of the load bearing stuff in an OS is much lower than it should be to justify the effort. This point is debatable for sure, but your arguments don't address it. | | |
| ▲ | staticassertion 4 hours ago | parent | next [-] | | > - It would be a big deal if Rust did have a safe dynamic linking ABI. Someone should do it. That's the main point I'm making. I don't think deflecting by saying "but C is no safer" is super interesting. I think we all agree that it would be a huge deal. > - So long as this problem isn't fixed, the upside of using Rust to replace a lot of the load bearing stuff in an OS is much lower than it should be to justify the effort. This point is debatable for sure, but your arguments don't address it. As you point out, this is the debatable part, and I'm not sure I get your justification here. | | |
| ▲ | pizlonator 3 hours ago | parent [-] | | This might end up being the forcing function (quoting myself from another reply in this discussion): > It can't be that replacing 20 C/C++ shared objects with 20 Rust shared objects results in 20 copies of the Rust standard library and other dependencies that those Rust libraries pull in. But, today, that is what happens. For some situations, this is too much of a memory usage regression to be tolerable. If memory was cheap, then maybe you could say, "who cares". Unfortunately memory isn't cheap these days | | |
| ▲ | staticassertion 2 hours ago | parent [-] | | How much overhead is that? Also, why would that have much overhead? Things deduplicate in memory. | | |
| ▲ | rlpb 2 hours ago | parent [-] | | Your apt update would still be huge though. When the dependency changes (eg. a security update) you’d be downloading rebuilds of 20 apps. For the update of a key library, you’d be downloading your entire distribution again. Every time. | | |
| ▲ | staticassertion 2 hours ago | parent [-] | | Oh, well yeah, statically linked binaries have that downside. I guess I don't think that's a big deal, but I could maybe imagine on some devices that are heavily constrained that it could be? IDK. Compression is insanely effective. |
|
|
|
| |
| ▲ | yxhuvud 5 hours ago | parent | prev | next [-] | | What you are asking for is to make a library definition replacement to .h-files that contain sufficient information to make rust safe. That is a big, big step and would be fantastic not only for rust but for any other language trying to break out of the C tar pit. | |
| ▲ | adgjlsfhk1 6 hours ago | parent | prev | next [-] | | The argument for unsafe ABI not being that big of a deal is that ABI boundaries often reflect organizational boundaries as well. E.g. the kernel wouldn't really benefit from a "safe ABI" because users calling into the kernel need to be considered malicious by default. | |
| ▲ | kittiepryde 6 hours ago | parent | prev | next [-] | | Would a safe ABI work with sandboxing the C code? I'm a bit unsure how one would construct a safe C ABI from Rust's side, | |
| ▲ | wang_li 6 hours ago | parent | prev | next [-] | | So you're calling for dynamic linking for rust native code? Because rust's safety doesn't come from runtime, it comes from the compiler and the generated code. An object file generated from a bit of rust source isn't some "safe" object file, it's just generated in a safe set of patterns. That safety can cross the C ABI perfectly fine if both things on either side came from rust to begin with. Which means rust dynamic linking. | |
| ▲ | 3 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | duped 6 hours ago | parent | prev [-] | | How could a safe dynamic linking API ever work? I think you're moving the goalposts significantly here. | | |
| ▲ | ninkendo 6 hours ago | parent | next [-] | | I don’t think GP is moving the goalposts at all, rather I think a lot of people are willfully misrepresenting GP’s point. Rust-to-rust code should be able to be dynamically linked with an ABI that has better safety guarantees than the C ABI. That’s the point. You can’t even express an Option<T> via the C ABI, let alone the myriad of other things rust has that are put together to make it a safe language. You can look to Swift for prior art on how this can be done: https://faultlore.com/blah/swift-abi/ It would be very hard to accomplish. Apple was extremely motivated to make Swift have a resilient/stable ABI, because they wanted to author system frameworks in swift and have third parties use them in swift code (including globally updating said frameworks without any apps needing to recompile.) They wanted these frameworks to feel like idiomatic swift code too, not just be a bunch of pointers and manual allocation. There’s a good argument that (1) Rust doesn’t consider this an important enough feature and (2) they don’t have enough resources to accomplish it even if they did. But if you could wave a magic wand and make it “done”, it would be huge for rust adoption. | | |
| ▲ | pizlonator 5 hours ago | parent [-] | | > I don’t think GP is moving the goalposts at all Thank you :-) > It would be very hard to accomplish. Yeah it's a super hard problem especially when you provide safety using the type system! The work the Swift team did here is hella impressive. > But if you could wave a magic wand and make it “done”, it would be huge for rust adoption. Yeah! |
| |
| ▲ | pizlonator 6 hours ago | parent | prev [-] | | > How could a safe dynamic linking API ever work? Fil-C solves it. I think Swift solves it, too. So it's solvable. No fundamental reason, that I know of, why Rust or any other safe language can't also have some kind of story here. > I think you're moving the goalposts significantly here. No. I'm describing a problem worth solving. Also, I think a major chasm for Rust to cross is how defensive the community gets. It's important to talk about problems so that the problems can be solved. That's how stuff gets better. |
|
|
| |
| ▲ | andrewflnr 5 hours ago | parent | prev [-] | | A safe ABI would be cool, for sure, but in the market (specifically addressing your prediction) I don't know if it's really that big a priority for adoption. The market is obviously fine with an unsafe ABI, seeing how C/C++ is already dominant. Rust with an unsafe ABI might then not be as big an improvement as we would like, but it's still an improvement, and I feel like you're underestimating the benefits of safe Rust code as an application-level frontline of security, even linked to unsafe C code. |
|
|
|
| ▲ | jjmarr 6 hours ago | parent | prev | next [-] |
| C++ ABI stability is the main reason improvements to the language get rejected. You cannot change anything that would affect the class layout of something in the STL. For templated functions where the implementation is in the header, ODR means you can't add optimizations later on. Maybe this was OK in the 90s when companies deleted the source code and laid off the programmers once the software was done, but it's not a feature Rust should ever support or guarantee. The "stable ABI" is C functions and nothing else for a very good reason. |
| |
| ▲ | pizlonator 6 hours ago | parent | next [-] | | I think if Rust wants to evolve even more aggressively than C++ evolves, then that is a chasm that needs to be crossed. In lots of domains, having a language that doesn't change very much, or that only changes very carefully with backcompat being taken super seriously, is more important than the memory safety guarantees Rust offers. | | |
| ▲ | jjmarr 6 hours ago | parent | next [-] | | In my view, this is a good thing. As a C++ developer, I regularly deal with people that think creating a compiled object file and throwing away the source code is acceptable, or decide to hide source code for "security" while distributing object files. This makes my life hell. Rust preventing this makes my life so much better. | | |
| ▲ | pizlonator 6 hours ago | parent [-] | | > As a C++ developer, I regularly deal with people that think creating a compiled object file and throwing away the source code is acceptable, or decide to hide source code for "security" while distributing object files. This makes my life hell. I mean yeah that's bad. > Rust preventing this makes my life so much better. I'm talking about a different issue, which is: how do you create software that's in the billions of lines of code in scale. That's the scale of desktop OSes. Probably also the scale of some other things too. At that scale, you can't just give everyone the source and tell them to do a world compile. Stable ABIs fix that. Also, you can't coordinate between all of the people involved other than via stable ABIs. So stable ABIs save both individual build time and reduce cognitive load. This is true even and especially if everyone has access to everyone else's source code | | |
| ▲ | jjmarr 4 hours ago | parent | next [-] | | > At that scale, you can't just give everyone the source and tell them to do a world compile. Stable ABIs fix that. Also, you can't coordinate between all of the people involved other than via stable ABIs. So stable ABIs save both individual build time and reduce cognitive load. Rust supports ABI compatibility if everyone is on the same compiler version. That means you can have a distributed caching architecture for your billion line monorepo where everyone can compile world at all times because they share artifacts. Google pioneered this for C++ and doesn't need to care about ABI as a result. What Rust does not support is a team deciding they don't want to upgrade their toolchains and still interoperate with those that do. Or random copy and pasting of `.so` files you don't know the provenance of. Everyone must be in sync. In my opinion, this is a reasonable constraint. It allows Rust to swap out HashMap implementations. In contrast, C++ map types are terrible for performance because they cannot be updated for stability reasons. | | |
| ▲ | pizlonator 4 hours ago | parent [-] | | My understanding: Even if everyone uses the same toolchain, but someone changes the code for a module and recompiles, then you're in UB land unless everyone who depends on that recompiles Am I wrong? | | |
| ▲ | at_compile_time 3 hours ago | parent [-] | | If your key is a hash of the code and its dependencies, for a given toolchain and target, then any change to the code, its dependencies, the toolchain or target will result in a new key unique to that configuration. Though I am not familiar with these distributed caching systems so I could be overlooking something. | | |
|
| |
| ▲ | kibwen 4 hours ago | parent | prev [-] | | > At that scale, you can't just give everyone the source and tell them to do a world compile. Firstly, of course you could. Secondly, you don't even need to, as NixOS shows. |
|
| |
| ▲ | Gigachad 3 hours ago | parent | prev | next [-] | | Isn’t this solution solved by just compiling your libraries with your main app code? Computers are fast enough that this shouldn’t be a huge issue. | |
| ▲ | SkiFire13 6 hours ago | parent | prev [-] | | C++ is still changing quite a lot though, just not in ways that fix the existing issues (often because doing so would break ABI stability). | | |
| ▲ | pizlonator 6 hours ago | parent [-] | | That is a reason why a lot of folks stick with C. In some sense, the chasm I'm describing hasn't been crossed by C++ yet | | |
| ▲ | pjmlp 6 hours ago | parent [-] | | Except as you well know, C might not change as fast, but it does change, including the OS ABI. Those folks think it doesn't. | | |
| ▲ | pizlonator 5 hours ago | parent [-] | | > Except as you well know, C might not change as fast, but it does change, including the OS ABI. I don't know that. Here's what I know: the most successful OSes have stable OS ABIs. And their market share is positively correlated with the stability of their ABIs. Most widely used: Windows, which has a famously stable OS ABI. (If you wanted to be contrarian you could say that it doesn't because the kernel ABI is not stable, but that misses the point - on Windows you program against userland ABIs provided by DLLs, which are remarkably stable.) Second place: macOS, which maintains ABI stability with some sunsetting of old CPU targets. But release to release the ABI provides solid stability at the framework level, and used to also provide stability at the kernel ABI level (not sure if that's still true - but see above, the important thing is userland framework ABI stability at the end of the day). Third place: Linux, which maintains excellent kernel ABI stability. Linux has the stablest kernel ABI right now AFAIK. And in userland, glibc has been investing heavily in ABI stability; it's stable enough now that in practice you could ship a binary that dynlinks to glibc and expect it to work on many different Linuxes today and in the future. So it would seem that OS ABIs are stable in those OSes that are successful. | | |
| ▲ | pjmlp 4 hours ago | parent | next [-] | | Speaking of Windows alone, there are the various calling conventions (pascal, stdcall, cdecl), 16, 32, 64 bits, x86, ARM, ARM64EC, DLLs, COM in-proc and ext-proc, WinRT within Win32 and UWP. Leaving aside the platforms it no longer supports. So there are some changes to account for depending on the deployment scenario. | |
| ▲ | anthk 5 hours ago | parent | prev [-] | | The most stable would be FreeBSD with compaNx libraries/modules for old binaries, where N = FreeBSD version number. |
|
|
|
|
| |
| ▲ | dpc_01234 3 hours ago | parent | prev | next [-] | | What's the stat of single-compiler version ABI? I mean - if the compiler guaranteed that for the same version of the compiler the ABI can work, we could potentially use dynamic linking for a lot of things (speed up iterative development) without committing to any long term stable API or going through C ABI for everything. | |
| ▲ | pjmlp 6 hours ago | parent | prev [-] | | The big question is does Rust want to play being adopted by those vendors, or it would leave them alone with languages that embrace native libraries. |
|
|
| ▲ | ahartmetz 6 hours ago | parent | prev | next [-] |
| Dynamic linking is also great for compile time of debug builds. If a large library or application is split up into smaller shared libraries, ones unaffected by changes don't need to be touched at all. Runtime dynamic linking has a small overhead, but it's several orders of magnitude faster than compile-time linking, so not a problem in debug builds. |
| |
| ▲ | PaulDavisThe1st 6 hours ago | parent [-] | | for developer turnaround time, it is huge. we explicitly do not statically link Ardour because as developers we are in the edit-compile-debug cycle all day every day, and speeding up the link step (which dynamic linking does dramatically, especially with parallel linkers like lld) is a gigantic improvement to our quality of life and productivity. | | |
| ▲ | elteto 6 hours ago | parent [-] | | A common pattern is dynamic linking for development and static linking for production-ready releases. | | |
| ▲ | PaulDavisThe1st 5 hours ago | parent | next [-] | | We considered doing both, but it turned out that the GUI toolkit we use was really, really not designed to be statically linked, so we stopped trying. | |
| ▲ | ahartmetz 5 hours ago | parent | prev [-] | | Yes, that's a good way to do it. |
|
|
|
|
| ▲ | eptcyka 6 hours ago | parent | prev | next [-] |
| The C ABI can already be used, it comes with all the existing safety guarantees that C will provide. Isn’t this as good as C? |
| |
| ▲ | pizlonator 6 hours ago | parent | next [-] | | It is as good as C. It's also as bad as C. I'm saying that the chasm to cross is a safe ABI. | | |
| ▲ | sagarm 6 hours ago | parent [-] | | There is no existing safe ABI, so this cannot be an adoption barrier. | | |
| ▲ | pizlonator 6 hours ago | parent [-] | | Lots of reasons why it is. I'll give you two. 1) It can't be that replacing 20 C/C++ shared objects with 20 Rust shared objects results in 20 copies of the Rust standard library and other dependencies that those Rust libraries pull in. But, today, that is what happens. For some situations, this is too much of a memory usage regression to be tolerable. 2) If you really have 20 libraries calling into one another using C ABI, then you end up with manual memory management and manual buffer offset management everywhere even if you rewrite the innards in Rust. So long as Rust doesn't have a safe ABI, the upside of a Rust rewrite might be too low in terms of safety/security gained to be worth doing |
|
| |
| ▲ | ozgrakkurt 6 hours ago | parent | prev | next [-] | | I found c ABI a bit too difficult in rust compared to c or zig. Mainly because of destructors. I am guessing c++ would be difficult in a similar way. Also unsafe rust has always on strict-aliasing, which makes writing code difficult unless you do it in certain ways. Having glue libraries like pyo3 makes it good in rust. But that introduces bloat and other issues. This has been the biggest issue I had with rust, it is too hard to write something so you use a dependency. And before you know it, you are bloating out of control | |
| ▲ | ahartmetz 6 hours ago | parent | prev [-] | | Not really. The foreign ABI requires a foreign API, which adds friction that you don't have with C exporting a C API / ABI. I've never tried, but I would guess that it adds a lot of friction. |
|
|
| ▲ | pjmlp 6 hours ago | parent | prev [-] |
| Indeed, Victor Ciura from Microsoft DevDiv has several talks on how this is currently an adoption problem at Microsoft. They have been working around it with DLLs, and COM/WinRT, but still the tooling isn't ideal. |
| |
| ▲ | rhdunn 5 hours ago | parent [-] | | COM is interesting as it implements interfaces using the C++ vtable layout, which can be done in C. Dynamic COM (DCOM) is used to provide interoperability with Visual Basic. You can also access .NET/C# objects/interfaces via COM. It has an interface to allow you to get the type metadata but that isn't necessary. This makes it possible to e.g. get the C#/.NET exception stack trace from a C/C++ application. |
|