| ▲ | rubiquity 10 hours ago |
| Matches my experience as well. Go fans have conflated "can easily make something concurrent" with "does concurrency well." Go's primitives for concurrency should almost never be used directly and Engineers below a certain skill level shouldn't be allowed to use them directly ever for long running production code. As another example, Go still has not yielded a correct implementation of Raft or Paxos while there are dozens in Java, C++, and Rust. Antithesis found some more bugs in HashiCorp's Raft implementation recently[0]. I'm sure etcd still has some kicking around. Maybe this is a "don't throw the baby out with the bath water' problem but the general evolution of Go has been lackluster. I reach for Rust, Zig, and modern Java instead depending on the specific needs and constraints. 0 - https://antithesis.com/blog/2026/finding-bugs-in-raft-implem... |
|
| ▲ | Thaxll 9 hours ago | parent | next [-] |
| The "world" runs on Kubernetes which is using Raft: https://pkg.go.dev/go.etcd.io/etcd/raft/v3 Are you saying that this implementation is wrong? "This Raft library is stable and feature complete. As of 2016, it is the most widely used Raft library in production, serving tens of thousands clusters each day. It powers distributed systems such as etcd, Kubernetes, Docker Swarm, Cloud Foundry Diego, CockroachDB, TiDB, Project Calico, Flannel, Hyperledger and more." One of the most popular distributed DB is Cockroach which is written in go and also uses Raft: https://github.com/cockroachdb/cockroach/tree/master/pkg/raf... |
| |
| ▲ | rubiquity 9 hours ago | parent | next [-] | | etcd has had numerous liveness and safety bugs, with one happening as recently as December of 2025. Would you consider that a correct implementation? You may be interested in knowing that the largest managed Kubernetes service in the world (AWS EKS) ripped out etcd for in favor of their homegrown consensus service for large scale EKS clusters: https://aws.amazon.com/blogs/containers/under-the-hood-amazo... | | |
| ▲ | fl0ki 8 hours ago | parent | next [-] | | etcd is some of the most amateur code I've ever seen, despite being one of the oldest and presumably most mature "infrastructure" projects written in Go. goBGP is arguably even worse. I don't have a third place in mind that's even worth mentioning relative to these two. | | |
| ▲ | alfons_foobar 6 hours ago | parent [-] | | just curious, what problems do you see with gobgp? (I have only a rather basic familiarity with go, but was considering gobgp for an infra project...) | | |
| |
| ▲ | Thaxll 8 hours ago | parent | prev | next [-] | | I'd like to know what you base your statement on that the Raft implementations in etcd or CockroachDB are incorrect. Your original paper does not mention those implementations, so where does that claim come from? | | |
| ▲ | gyesxnuibh 4 hours ago | parent [-] | | Having run a fleet of 100s of etcd clusters for 10000s of rps, and the fact that upstream runs tests similar to antithesis and recently partnered with antithesis [0], and jepsen has tested it long ago as well [1]. Etcd's raft algorithm is fine. Someone even did a TLA+ proof on it in the last couple years[2]. Yes there was a correctness issue a few years ago but otherwise the person you're replying to doesn't know what they're talking about. Also those bugs have nothing to do with the raft implementation, but instead the state machine implemented on top. 0: https://etcd.io/blog/2025/autonomus_testing_with_antithesis/ 1: https://jepsen.io/analyses/etcd-3.4.3 2: https://github.com/etcd-io/raft/pull/113 | | |
| ▲ | iscoelho an hour ago | parent | next [-] | | "there was a correction issue" is downplaying it. Etcd is truly the worst example of Raft. Etcd corruption and loss of quorum is extremely common in practice and the GitHub issues sit for years. The design is simple, the performance is modest, yet it still has still never been reliable, despite being marketed as so. I can't speak to whether this is specifically due to their Raft implementation, but I'd argue the entire codebase is over-engineered and questionable. | | |
| ▲ | AlphaSite 12 minutes ago | parent | next [-] | | My beef with etcd is that its neither performant nor reliable. Its very much {reliable, performant, flexible} pick none. | |
| ▲ | gyesxnuibh an hour ago | parent | prev [-] | | Their lock, leader election, sessions, and leases are all awful and I'd never recommend anyone to use those. But as a strongly consistent kv store and if you need the watch mechanics, its useful. It has its place and that's mostly being used by kubernetes. |
| |
| ▲ | cobbzilla 2 hours ago | parent | prev [-] | | Is the correctness of its implementation of the algorithm unaffected by bugs in this state machine? Maybe I missed something. | | |
| ▲ | gyesxnuibh an hour ago | parent [-] | | The raft algorithm works and if you implemented a less complex state machine (like using a simpler kv store that doesn't need global event ordering via revisions and watches) it would work. That's what antithesis said they did to test the raft algorithms in the other article linked |
|
|
| |
| ▲ | camkego 8 hours ago | parent | prev [-] | | Sorry to pile on, but yeah, I wanted to use etcd during 2021 and 2022, around v3.5, but etcd had serious issues including silent data corruption. If you are curious, ask gemini flash "there were a number of etcd releases years ago where it seems a new wave of developers came in and started breaking everything" |
| |
| ▲ | ablob 3 hours ago | parent | prev | next [-] | | Surely the King is doing it, so that must be the correct way. Look, the King even wears clothes and is totally not naked at all. That the world runs on Kubernetes is no qualitative statement about the correctness of its Raft implementation. You can say that it's clearly good enough to not matter most of the time, but that is a different statement.
No matter who you look at, they're just cooking with gas like you do, and they can make mistakes in just the same way. Now; I'm only attacking your argument. I do neither know nor particularly care about the correctness of that implementation itself. There's been better refutations of the claim you replied to in other answers anyway. | |
| ▲ | formerly_proven 8 hours ago | parent | prev | next [-] | | Now there's three of them https://github.com/hashicorp/raft | |
| ▲ | aksss 9 hours ago | parent | prev [-] | | That's not remotely what he's saying at all. | | |
| ▲ | volkk 9 hours ago | parent | next [-] | | > As another example, Go still has not yielded a correct implementation of Raft or Paxos > are you saying this implementation is wrong? > That's not remotely what he's saying at all. I'm v confused by this thread | |
| ▲ | quietbritishjim 9 hours ago | parent | prev | next [-] | | I don't care for Go myself (especially its concurrency model, which is a total dinosaur in a world where we have structured concurrency) so I'm not saying this to support my favourite language, but: That is literally what the comment says. | |
| ▲ | rubiquity 9 hours ago | parent | prev [-] | | Thank you. I don't know why this is so complicated. |
|
|
|
| ▲ | HAL3000 8 hours ago | parent | prev | next [-] |
| > Go is bad so "I reach for Rust, Zig..." I hope my every competitor will take your advice to heart, as one of our competitors did when they read that "Go is not a memory safe language", so they wrote a blog about how they are porting to Rust. While our team was moving fast and using those "primitives that should almost never be used" around our long running production code base with success. Some time has passed and now their company does not exist anymore and we have a lot of their clients. Thank you! |
| |
| ▲ | xvedejas 2 hours ago | parent | next [-] | | Where would one ever read that Go is not memory safe? That's just a false claim, and anyone believing it would have probably gone out of business regardless of choice of programming language. | | |
| ▲ | ReactiveJelly an hour ago | parent | next [-] | | It looks like Go is memory safe for single threads and channels, but not for shared memory between threads: https://en.wikipedia.org/wiki/Go_(programming_language)#Lack... > Go's internal data structures like interface values, slice headers, hash tables, and string headers are not immune to data races, so type and memory safety can be violated in multithreaded programs that modify shared instances of those types without synchronization.[113][114] | |
| ▲ | gpm an hour ago | parent | prev | next [-] | | By a strict definition of memory safety it isn't - you can tear two-pointer-wide values using data races and cause arbitrary memory issues if you try to using only normal code. It's close enough for most purposes... but it isn't. | |
| ▲ | pstuart 2 hours ago | parent | prev [-] | | My ex-boss was a JS guy and then moved over to Rust. He loathed Go because it has pointers and it's possible to use a nil pointer if you are not competent. JS is fine for what and where it is, Rust is fine too. I just appreciate the stupid simple nature of Go and it does the job just fine. |
| |
| ▲ | sunrunner 3 hours ago | parent | prev | next [-] | | Perhaps that company failed because it chose to port things to Rust and not because of Rust itself? Or any other number of reasons that survivorship bias might be mistaking. | |
| ▲ | za3faran 4 hours ago | parent | prev [-] | | What domain is your company in? |
|
|
| ▲ | jerf 8 hours ago | parent | prev | next [-] |
| You seem to be implying, based on the rest of the thread, that Go has some sort of special defect that keeps it from implementing Raft correctly. But the "special defect" that Go has is that it in practice implements the same primitives in practice that almost every other mainstream language does, rather than implementing some sort of super-safe concurrency primitive like Erlang or Pony, or being immutable like Haskell. And even those things are of only marginal utility for Raft, preventing some local issues, but the hard part of Raft is more in the logic and the communication, for which none of these languages have any sort of special support or anything that will particularly help you get it right. Of the languages you listed only Rust provides any assistence over the standard mainstream languages, and like I said, in the context of Raft, it is not necessarily all that helpful. If you want to see something that could potentially impact Raft's correctness, search the last couple of days of the HN front page for choreographic languages [1]. But none of these are even remotely mainstream enough to depend on for anything. Nor do I know if anyone in these languages has implemented Raft. A rather good test case for them, if any of them are looking. That's something that could actually help a Raft implementation's correctness, not just fiddle around the edges of local concurrency issues. [1]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... |
| |
| ▲ | lokar 4 hours ago | parent [-] | | Also, wasn’t this about concurrency? You could, if you really wanted, write a paxos or raft implementation with no concurrency. |
|
|
| ▲ | evil-olive 17 minutes ago | parent | prev | next [-] |
| > Go still has not yielded a correct implementation of Raft or Paxos while there are dozens in Java, C++, and Rust. Antithesis found some more bugs in HashiCorp's Raft implementation recently[0]. the source you link to contradicts your own claims. they say: > we’ve found bugs in every Raft implementation we’ve tested, including HashiCorp Raft, Aeron Cluster, OpenRaft, and MicroRaft (besides Go, that's 2 in Java and 1 in Rust) |
|
| ▲ | ramoz 10 hours ago | parent | prev | next [-] |
| That does not seem like a fair/accurate reference? The antithesis author states: "we’ve found bugs in every Raft implementation we’ve tested, including HashiCorp Raft, Aeron Cluster, OpenRaft, and MicroRaft"
|
| |
| ▲ | rubiquity 10 hours ago | parent [-] | | You're misreading what I said. I didn't say other languages don't have buggy Raft/Paxos implementations, just that Go is yet to yield a single correct one. | | |
| ▲ | jibal 9 hours ago | parent [-] | | I think you're projecting. You wrote > Go still has not yielded a correct implementation of Raft or Paxos while there are dozens in Java, C++, and Rust. That says that there are correct (i.e., bug-free) implementations in those languages. The GP noted > "we’ve found bugs in every Raft implementation we’ve tested, ..." which says that there aren't any correct ones. You then wrote > I didn't say other languages don't have buggy Raft/Paxos implementations which is a strawman. The issue is whether there are correct implementations. That there are buggy ones is irrelevant. (FWIW I have no dog in this fight ... I'm just reading here.) | | |
| ▲ | rubiquity 9 hours ago | parent [-] | | The intersection of the set of Raft libraries Antithesis tested and all Raft libraries in existence do not fully overlap. I personally have worked on multiple proprietary ones that Antithesis would not have access to. | | |
| ▲ | _dain_ 6 hours ago | parent | next [-] | | I work at Antithesis, we're happy to test out any Raft implementation that has so far escaped our notice :) | | |
| ▲ | rubiquity 3 hours ago | parent [-] | | I love what you all do! I don’t have any to submit. Enough time with consensus teaches you to avoid it unless it’s really, truly needed. |
| |
| ▲ | 0x696C6961 4 hours ago | parent | prev | next [-] | | Lol "I swear have a girlfriend, she just goes to a different school" | |
| ▲ | jibal 9 hours ago | parent | prev [-] | | Even if so, the "You're misreading what I said" charge was bogus and it would be nice if you admitted that. Edit: > What are they implying by citing that? That Raft implementations in all languages have bugs? That's what it says. > I've already pointed out that is false. You claimed that, and it's being disputed. > Please let me know, since you're so comfortable speaking for them. This has veered into bad faith ... I won't comment further. | | |
| ▲ | overfeed 9 hours ago | parent | next [-] | | Further, they can still edit their comment to correct it, but opting not to. | |
| ▲ | rubiquity 9 hours ago | parent | prev [-] | | But they are misreading what I said. My original post is clearly about Go. What they wrote is also ambiguous. > The antithesis author states: > "we’ve found bugs in every Raft implementation we’ve tested, including HashiCorp Raft, Aeron Cluster, OpenRaft, and MicroRaft" What are they implying by citing that? That every language has a Raft implementation with bugs? Yes that's probably accurate because lots of people make Raft implementations for fun and learning. Again, Go does not have a single Raft/Paxos implementation that is rock solid. I have seen many in C++, Java, and Rust that are doing tens of millions of requests per second in production for over a decade. Is their point that Go is not the only language with this problem? My post already points out the track record is that Go is the problem for writing correct code in highly critical domains. | | |
| ▲ | Dylan16807 8 hours ago | parent | next [-] | | If we rely on their evidence alone, it suggests nobody has ever made a correct implementation, so we learn nothing about Go. "Go has yet to yield a correct one" is an extremely misleading way to present evidence that says the same thing about every language. The only way this becomes useful for comparing languages is if somebody gives evidence of correct implementations in other languages. You're claiming they exist but with no evidence and suggesting they're secret. How do you know those don't have bugs? Did any concurrency bug experts do extensive testing on them? And can we disprove secret Go implementations of the same quality? | |
| ▲ | FireBeyond an hour ago | parent | prev | next [-] | | You're being very mealy-mouthed, even here, it reads as "The pre-eminent Go implementation can't even get it right" and yet the implementations Antithesis tested in Rust are apparently people's random "fun and learning" projects, nothing serious, and certainly not all the proprietary implementations that you've used that are all correct. | | |
| ▲ | jibal 28 minutes ago | parent [-] | | Also, it simply isn't true that @ramoz misread what they wrote, as I pointed out in both of my comments. |
| |
| ▲ | joshuamorton 8 hours ago | parent | prev [-] | | > That every language has a Raft implementation with bugs? Yes that's probably accurate because lots of people make Raft implementations for fun and learning. Again, Go does not have a single Raft/Paxos implementation that is rock solid. I have seen many in C++, Java, and Rust that are doing tens of millions of requests per second in production for over a decade. No, they are citing that every Raft implementation that Antithesis has tested has bugs. The etcd implementation you note in go that has bugs also does tens of millions of QPS and is over a decade old. How are you confident that the proprietary implementations that presumably haven't been fully tested don't have subtle bugs that don't show up in practice? |
|
|
|
|
|
|
|
| ▲ | tptacek 15 minutes ago | parent | prev | next [-] |
| The Raft bugs are the wrong kind of concurrency --- they're distsys bugs, not multithreading bugs. Not a good example, and a little telling that you'd cite it. |
|
| ▲ | hintymad 9 hours ago | parent | prev | next [-] |
| Java has so many excellent concurrency containers, plus robust 3rd-party containers like JCTools. It puzzles me why Go communities do not offer such containers. |
| |
| ▲ | Groxx 3 hours ago | parent [-] | | No thread/goroutine handles for fork/join handling from "outside", and no generics for many formative years that influenced tons of habits, then significantly weaker generics (improving very soon[1]), have all led to most concurrent code to be very "intrusive" - you create bare threads and add bare synchronization primitives (or nearly) by hand inside the threaded code to make it concurrent. `errgroup` is as far as a lot of code goes, in terms of sophistication. Java leans heavily in the other direction: a lot of concurrency is added externally, without changing existing code, often in very declarative-flavored ways. E.g. Future<T> serves as a foundation for a ridiculous amount of stuff, while Go forces channels for `select` whether they model your problem nicely or not, and they're very difficult (often impossible) to wrap without changing semantics. There are very obviously lots of counter-examples for both langs (`synchronized`, rill in Go, etc), and I expect Go to become more Java-flavored in time (it already has moved this direction somewhat, and 1.27 will enable a lot more). But I think it's a fair summary of broad ecosystem habits. 1: https://tip.golang.org/doc/go1.27 (not yet released) |
|
|
| ▲ | mrsilencedogood 9 hours ago | parent | prev | next [-] |
| To combine both TFA with this comment: I find that LLMs are ~fine at generating/editing gocode, or at least as ~fine as they generate most mainstream languages. But good god, the second it gets to anything concurrency-related, it just loses its mind. As much as it's gotten vaguely ok to try to let the agents loose on some bits of the codebase, they simply can't even do table stakes stuff with the kinds of concurrency you see in real life. |
| |
| ▲ | nasretdinov 7 hours ago | parent | next [-] | | Correct concurrent code is mind-bogglingly hard even for seasoned veteran humans (and don't get me started on distributed programming...), so it's hardly surprising that LLMs with their limited context windows into the code have a hard time writing correct concurrent code | |
| ▲ | rubiquity 8 hours ago | parent | prev | next [-] | | My experience as well. LLMs also struggle with Rust's many abstractions and offerings but you can know that if it compiles it is data race free and work with the LLM to use better abstractions over time. Zig is also good at this but requires more up front design (thread-per-core, static allocation, etc.) and consistent checks to verify rules are followed. | | |
| ▲ | bb88 7 hours ago | parent [-] | | C/C++ has "compiles but may have undefined behavior". Golang has numerous "compiles but has incorrect behavior" (normally known as footguns). Meanwhile with Rust, if you get past the compilation step, bugs become much much fewer. (You can still have memory leaks, but those are easily traceable). It seems like claude code can code Rust pretty well with Opus, and I've started moving codebases away from Golang to Rust at work with Opus. Spin up an LLM and it cranks on it for a while, and as a benefit, I get easy apis to build on with other languages. And that's the problem with Golang really, not that it's a bad language per se (all languages have footguns), but that the language interoperability story is terrible. Meanwhile Rust and Python/C/C++ go great together like peanut butter and chocolate. And I love it. | | |
| ▲ | maleldil 2 hours ago | parent [-] | | > You can still have memory leaks And deadlocks. "Fearless concurrency" helps a lot, but logic bugs are still possible. |
|
| |
| ▲ | za3faran 4 hours ago | parent | prev [-] | | I wouldn't be surprised if Java has a much better experience here. After all, java.util.concurrent has many great implementations, and Java's `record`s are immutable, as are it's upcoming value types. | | |
|
|
| ▲ | a2ff6eeb0 4 hours ago | parent | prev | next [-] |
| Yeah, but AI is better at debugging than people are, so what's the issue? |
|
| ▲ | jeffbee 8 hours ago | parent | prev [-] |
| What primitives are we discussing? Any Go programmer can and should use the `go` keyword and the `sync.Mutex` type from their first program. |
| |
| ▲ | closeparen 7 hours ago | parent [-] | | It's pretty easy to get yourself into trouble with channels: deadlocks, send on closed, channel leaks, deadlocks "fixed" thoughtlessly with arbitrarily-sized buffers, etc. | | |
| ▲ | jeffbee 7 hours ago | parent [-] | | This seems to have little to do with Go's facilities. Any concurrent program has hazards like these. | | |
| ▲ | treyd 34 minutes ago | parent | next [-] | | Rust's concurrency libraries leverage the type system to make these issues much harder to encounter. | |
| ▲ | closeparen 7 hours ago | parent | prev [-] | | In my experience, shared memory instills the appropriate fear and caution, while the apparent simplicity of channels encourages novice Go programers to take on concurrency projects beyond their abilities and without due care. Been on both the submitter and reviewer side of that plenty of times in 10 years. |
|
|
|