> Databricks’ multi-million line codebase

The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?

▲

trollbridge 2 hours ago | parent | next [-]

Because if they're like previous places I worked out with proprietary commercial codebases, the code is old, has been around a long time, and trying to shrink it is not a priority.

It's a good stress test for the LLM because it is not an "ideal" codebase.

▲

alexott an hour ago | parent | prev | next [-]

:facepalm: "orchestration/management layer" that just runs dozens of millions of VMs per day, across the three clouds, with things like model serving, AI gateway, OLTP (Lakebase/Neon), Unity Catalog, etc.

Forget Databricks == Apache Spark...

▲

ozgrakkurt 3 hours ago | parent | prev | next [-]

Old codebase, you always add code and never remove it. So it is expected to be like this.

Deleting code is difficult and almost never makes sense afaik

	▲	wwind123 2 hours ago \| parent \| next [-]
		On one hand, I understand that some old code is hard to delete because it's hard to detangle a lot of the legacy dependency. On the other hand, too much useless old code existing in the code base by itself could become a big maintenance burden for both humans and AI. In some cases at some point it might become more economical to just invest a bunch of resource to detangle the dependencies to be able to remove the old code.
	▲	tijs 2 hours ago \| parent \| prev [-]
		Every line you delete is a line you no longer need to maintain. We aggressively prune old code in our apps and it has definitely helped with maintainability. For a mobile app it’s also code you don’t ship so that’s a nice bonus which I guess is not much of an argument on a backend codebase

▲

appplication 3 hours ago | parent | prev [-]

LoC isn’t a super helpful metric so I think the better question is why is the headline using it. I can say I’ve personally created about 200k LoC code in the last 5 years and most of that has some value. But it really doesn’t say might about how much value or really anything else meaningful.

	▲	InsideOutSanta 2 hours ago \| parent [-]
		They're probably mentioning the size of the code base as an indicator for how difficult it is for an LLM to understand where and how to make changes.