I'm not sure what you mean by "running a job that ran a worker". Running the system is vastly different than building and maintaining one. I work on highly available and distributed systems spanning multiple data centers, and single node scalability of database kernels, so I guess I could also say that I've done and still doing work at the highest level but in my case it is the core development. Which one is yours?

▲

dekhn 12 hours ago | parent [-]

The system I built was called exacycle. It ran as a borg job in every cluster we had access to, with priority 0 or 1, with N tasks (where N=# of machines in the cluster). Borg ran the "worker" task as long as there were resources free on the machine (often, several workers ran on the same machine, and no workers ran on other machines). There was a global schedulers that picked mini-tasks off a global queue and fed them to the cluster-level scheduler (batches based on instant resource availability, and subject to some data locality optimization) which then fed them to the machine workers.

At peak, we had 1-3 million xeon cores running our tasks (protein folding, protein design, drug docking), 24/7. This was before GPUs were really a thing. Each mini-task was basically a rosetta or gromacs or autodock binary running on local inputs and writing local outputs for ~1 hour using ~1GB RAM and no network. Input and output was via colossus, or distributed through a global torrent network (literally all 1-3 million cores formed a single data transfer group, with data passed between clusters at low probability).

I built most of the code of the system (which depended on Google infra) including the scheduler and the machine worker, and also ran the system as on-call "SRE" for 3 years. During that time, we nearly almost didn't take down the Google global network, and had small latency impacts on many services. Managing to run the system for so long without antagonizing ads or search or youtube was a major challenge.

I would say the complexity of the system I built (considering everything down to the OS) was roughly the level of a few human cells in a plate, in terms of interacting components and the complexity of their interactions.

Here are some of the publications that came from running exacycle (we provided strong evidence that using more compute improves outcomes in molecular simulations)

https://pubmed.ncbi.nlm.nih.gov/24345941/ https://pubmed.ncbi.nlm.nih.gov/24265211/

▲

menaerus 11 hours ago | parent [-]

I appreciate the details, very interesting. So from a quick glance my understanding is that exacycle is a task partitioning and dispatching system built on top of existing google infra runtime (borg). Tasks are units that are massively parallel and distributed across the compute, and have no or almost no shared data. Problem space seems to be built around self contained tasks which have well defined input and output, and main engineering challenge in the exacycle itself would be the optimizing the system for maximum resource utilization? It's complex but not as complex as open problem space engineering problems dealing with highly concurrent and distributed workloads with shared state and that are unknown at the time you're designing the algorithm.

Anyway, my point was not in that I would equate software complexity with the complexity found in biology but that I see the same trend and similar type of challenges that can be found in both fields and therefore I see both fields amenable to the AI advancements whereas many people from the thread do not, and I disagree with that.

▲

dekhn 11 hours ago | parent | next [-]

Yes, we explicitly avoided any kind of high concurrency/parallelism. That was an intentional decision, although note we did have a global torrent network across all the workers, whose topology (node interactions) formed an extremely complex graph.

I used to work in the MPI world on supercomputer jobs and it was miserable. I also worked on other high concurrency systems at Google (helped run the Ads DB, did training and engineering of Sibyl). I continue to insist these systems are incredibly simple compared to even the most basic human cells interactin g.

▲

menaerus 10 hours ago | parent [-]

Well, we have seen AlphaFold solving a decades old protein folding problem. There's no reason for me to believe, with all the AI advancements I've seen in the meanwhile, that we will not solve other problems in this field too.

	▲	dekhn 8 hours ago \| parent [-]
		AlphaFold didn't solve a decades old protein folding problem. They substantially improved protein structure prediction but with important limits. THe primary bottleneck for a drug is getting it through clinical trials. No matter how much you speed up lead detection, development, and in vivo studies, the bottleneck will still be clinical trials. Now, if we could make lead detection better at generating molecules that get through clinical trials, that's great- but it would take decades to know for sure. If we could make development less likely to produce toxic molecules that trigger the immune system, sure that's great- but again, we're talking decades before we know if it worked. Attempting to improve clinical trials- well, those are already fairly well designed and run, but it's likely you could make improvements there too that increased efficacy. But again, you're still just dealing with the FDA, filing tons of paperwork, and tons of people looking at the paperwork. Unless you fix those systematic issues, we're not going to accelerate drug approvals.

▲

randomImmigrant 10 hours ago | parent | prev [-]

I definitely see a huge role for AI in the lab. Don’t get my wrong at all. The physical labor in the biology lab would go a lot smoother if I knew the data analysis pipelines could be handled with AI. I shudder to think where my phd would have gone if I’d had AI. And having had to work with jury rigged microscopy rigs and patch clamping rigs, I’m fascinated to see where physical AI for actuators and control systems goes. Certainly I’d be using Claude to improve the control UX to my taste so I could wander my samples with greater control, save time and reduce error.

But I don’t see how any current AI system will have the curiosity and drive. You need to create conditions for meaningful experiments to be done and a mechanistic way of doing this doesn’t work. The most elegant experiments find ways to cut to the core of the emergent behavior, around all the simultaneous changes happening in different levels of the system from which said behavior arises.

You simply cannot see it all. Not every AI can, we are measurement limited. We can track and measure some dynamics, and the art of the experiment lies in thinking through the connections, then picking the right places to look to catch the system in the act. And you better do it so the failure guides the next experiment, and the next and the next, till you finally have sliced through the complexity and found some key nodes that hold the keys.

And usually after all that, you find your key node is regulated by the key nodes of a few other grad students. The complexity here is in part because the entire system is in constant conversation across levels of organization. The causal flow is looped and is also very sensitive to the environment.

AI has a big role to play, but I doubt it’ll parse it without humans at play. It simply cannot invent a slight modification of a microscope and physically connect the new lens element in the right place, calibrate it, all in time for the biology grad student next door to bring some cells to image that are a mix of 40 days old (no contamination! Magic!) vs 30, 20, and so on, to do a very expensive experiment (in terms of labor and time and rats and materials).

So much can go wrong on so many vectors. Which is also what pushes humans to their creative best. And AI will help a lot. But till you have a robot that can drive, but also do mouse dissections and staining and manipulate a microscope with faulty parts to get a clear image, you’re going to need humans to study biology.

When you invent these super robots that somehow slide through all this physical labor without consuming a whole lot of power, we can talk.

Meanwhile, at the cutting edge, where neither procedure nor mass produced machinery exists to do the whole thing, the reality is, we already use a lot of fancy tech in biology, and can use more, and AI can help keep the focus on the actual physical experiments that take so much care and attention.