What i take away is the simplicity and scaling behavior. The ML field often sees an increase in module complexity to reach higher scores, and then a breakthrough where a simple model performs on-par with the most complex. That such a "simple" architecture works this well on its own, means we can potentially add back the complexity again to reach further. Can we add back MSA now? where will that take us?

My rough understanding of field is that a "rough" generative model makes a bunch of decent guesses, and more formal "verifiers" ensure they abide by the laws of physics and geometry. The AI reduce the unfathomably large search-space so the expensive simulation doesn't need to do so much wasted work on dead-ends. If the guessing network improves, then the whole process speeds up.

- I'm recalling the increasingly complex transfer functions in redcurrant networks,

- The deep pre-processing chains before skip forward layers.

- The complex normalization objectives before Relu.

- The convoluted multi-objective GAN networks before diffusion.

- The complex multi-pass models before full-convolution networks.

So basically, i'm very excited by this. Not because this itself is an optimal architecture, but precisely because it isn't!

Correct. For those that might not follow, the MSA is used to generalize from known PDB structures to new sequences. If you train on AlphaFold2 results, those results include that generalization, so that your model no longer needs that capability (you can rely on rote memorization). This simple conclusion seems to have escaped the authors.

Is this so unusual? Almost everything that is simple was once considered complex. That's the thing about emergence, you have to go through all the complexities first to find the generalized and simpler formulations. It should be obvious that things in nature run off of relatively simple rulesets, but it's like looking at a Game of Life and trying to reverse engineer those rules AND the starting parameters. Anyone telling you such a task is easy is full of themselves. But then again, who seriously believes that P=NP?

And AlphaFold was validated with experimental observation of folded proteins using X-rays

As I understand it, folding at home was a physics based simulation solver, whereas alphafold and its progeny (including this) are statistical methods. The statistical methods are much, much cheaper computationally, but rely on existing protein folds and can’t generate strong predictions for proteins that don’t have some similarities to proteins in their training set.

In other words, it’s a different approach that trades off versatility for speed, but that trade off is significant enough to make it viable to generate protein folds for really any protein you’re interested in - it moves folding from something that’s almost computationally infeasible for most projects to something that you can just do for any protein as part of a normal workflow.

Yep, that and SETI@Home. I loved the eye candy, even if I didn't know what it fully meant.

Apparently from a F@H blog post [1] they say it's still useful to know the dynamics of how it folded, in addition to the final folded shape. And that having ML-folded proteins is a rich target for simulation to validate and to understand how the protein works

[1] https://foldingathome.org/2024/05/02/alphafold-opens-new-opp...

They're still going and have made some great discoveries over the years.

https://foldingathome.org/papers-results/?lng=en

I contributed a lot on there too used my 3080Ti-FE as a small heater in the winter

Team F@H forever!

And the abstract alone says (if I'm reading it correctly), "It still takes AI; just not nearly as much as others are doing."

For those interested in the GitHub link.

https://github.com/apple/ml-simplefold

Apple has an ML research group. They do a mixture of obviously-Apple things, other applications, generally useful optimizations, and basic research.

https://machinelearning.apple.com/

No idea, but can I be signed up for R&D jobs where you don't necessarily build something generating revenue?

Maybe these are just projects they use to test and polish their AI chips? Not sure.

To sell computers? 20 years ago, Apple had scientific poster sessions at WWDC and worked to bring PyMol to the Mac. The pictures of proteins you see in the paper were generated with PyMol as are probably >50% of the protein images in scientific papers for the last 15 years.

Local inference. I imagine they have an interest in making this and other cutting edge models small enough to be possible to do quick inference on their desktop machines. The article shows that, with Figure 1E demonstrating inference on an M2 Max 64 GB.

Frankly, it's a great idea. If you are a small pharma company, being able to do quick local inference removes lots of barriers and gatekeeping. You can even afford to do some Bayesian optimization or RL with lab feedback on some generated sequences.

In comparison, running AlphaFold requires significant resources. And IMHO, their usage of multiple alignments is a bit hacky, makes performance worse on proteins without close homologs, and requires tons of preprocessing.

A few years back, ESM from Meta already demonstrated that alignment-free approaches are possible and perform well. AlphaFold has no secret sauce, it's just a seq2seq problem, and many different approaches work well, including attention-free SSMs.

This may not be the actual reason in this case, but I think it's good to be aware of: A non-zero chunk of "ai for science" research done at tech companies is basically done for marketing. Even in cases where it's not directly beneficial for the companies products or is unlikely to really lead to anything substantial, it is still good for "prestige"

Reputation laundering?

Probably because ByteDance and Facebook (spun out into EvolutionaryScale) are doing it

They're jealous they haven't won a Nobel prize

Prowlly cuz Siri didn’t work out

How do you call the opposite of green washing? When you want to show that you are burning as much energy on training models as the others.

Well, figure a) shows a ribbon representation of the fold (as helices and strands) of the protein 7QSW (https://www.ebi.ac.uk/pdbe/entry/pdb/7qsw) which is RubisCO (https://en.wikipedia.org/wiki/RuBisCO), an plant protein that plays a key role in photosynthesis.

The different colours are for the predicted and 'real' (ground truth) models. The fact that it is hard to distinguish is partly the - as you point out - weird colour choice, but also because they are so close together. An inaccurate prediction would have parts that stand out more as they would not align well in 3D space.

I’m not sure AF3’s performance would hold up if it hadn’t been trained on data from AF2 which itself bakes in a lot of inductive bias like equivariance

Their representation is simpler, just a transformer. That means you can just plug in all the theory and tools that have been developed specifically for transformers, most importantly you can scale the model easier. But more than that, I think, it shows that there was no magic to AlphaFold. The details of the architecture and training method didn't matter much. All that was needed was training a big enough model on a large enough dataset. Indeed lots of people who have experimented with AlphaFold have found it to behave similiar to LLMs, i.e. it performs well on inputs close to the training dataset and but it doesn't generalize well at all.

It may be a change in future models, perhaps. Here's one person's opinion:

https://genomely.substack.com/p/simplefold-and-the-future-of...

But as with anything in research, it will take months and years to see what the actual implications are. Predictions of future directions can only go so far!

No. AlphaFold doesn't do dynamics; it does end-state snapshots only. It does not do anything about the motion of the atoms, which is the core functionality of MD.

I was curious about what was released, and the parameters for AlphaFold V3 are only given to certain groups for non-commercial use: https://github.com/google-deepmind/alphafold3?tab=readme-ov-...

However, it seems like anyone can download the parameters for AlphaFold V2: https://github.com/google-deepmind/alphafold?tab=readme-ov-f...

MD was never really a viable way to do structure prediction, so it didn't become obsolete with AlphaFold. Instead, MD is more useful for studying the physical process of protein folding (before the protein folds to its final structure, as well as once it has reached its final structure and sort of jiggles and wiggles around that).

> it's just running a simulation of your protein surrounded by some water molecules for the same number of nano-seconds nature do.

No, the environment is important. Also, some proteins fold while being sequenced.

Folding can also take minutes in some cases, which is the real problem.

> which is a complex problem involving Quantum Mechanics

Most MD simulations use classical approximations, and I don't see why folding is any different.

Its not an LLM, It's a transformer. I know the terms are really being butchered in media, but if we're gonna use the term LLM instead of AI, we better make sure it's actually a "large language model" that is being refereed to. If you're unsure, call it a neural net, or machine learning algorithm, or AI.

It's indeed a large model. But if you knew the history of the field, it's a massive improvement. It has progressed from a almost "NP" problem only barely approachable with distributed cluster compute, to something that can run on a single server with some pricey hardware. The smallest model is only here is only 100M parameters and the largest is 3B parameters, that's very approachable to run locally with the right hardware, and easily within the range for a small biotech lab (compared to the cost of other biotech equipment)

It's also (i'd argue) one of the only truly economically and sociably valuable AI technologies we've found over the past few years. Every simulated protein fold is saving a biotech company weeks of work for highly skilled biotech engineers and very expensive chemicals (In a way that that truly only supplement rather than replace the work). Any progress in the field is a huge win for society.

Folding proteins is pretty valuable and this model is comparably small

This doesn't seem like particularly wasteful overinvestment.

Granted, I'm more excited about the research coming out of arc

How simple did you think it was before?

I guess it's because SimpleFold came from a research lab with different autonomy and less competing interests and internal politics...

I am genuinely interested where the strong negativity towards Siri has come from in recent culture. From what I gather it's likely due to the high expectations we have for Apple. But what I don't really get is why is there not a similar amount of negativity being directed at Google or Samsung, who both have equally shit phone AI assistants (obviously this is just from my perspective, I am a daily user of both iOS and a Samsung Android)

I am not trying to defend Apple or Siri by any means. I think the product absolutely should (and will) improve. I am just curious to explore why there is such negativity being directed specifically at Apple's AI assistant.

Something like this doesn’t actually have to work. There were no expectations at all in this space.

Meanwhile, people expect perfection from Siri. At this point a new version of Siri will never live up to people’s expectations. Had they released something on-par with ChatGPT, people would hate it and probably file a class action lawsuit against Apple over it.

The entire company isn’t going to work on Siri. In a large company there are a lot of priorities, and some things that happen on the side as well. For all we know this was one person’s weekend project to help learn something new that will later be applied to the priorities.

I’ve made plenty of hobby projects related to work that weren’t important or priorities, but what I learned along the want proved extremely valuable to key deliverables down the road.

As I understand it, Siri and Alexa could be plugged into an LLM, but changing it to an "open world" device that can tell your kid something disturbing, text all of your contacts, buy groceries, etc. comes with serious risk of reputational harm. While still falling short of people's expectations if it isn't ChatGPT-quality. OpenAI is new enough that they get to play by different rules.

Fair point, even X was able to pump out a usable AI, grok.

Should an entry in a field preclude other ones. I encourage you to apply reductio-ad-absurdum here. Should Pepsi exist if Coke does? Should C exist if Fortran does?

Maybe they've been working on it, but got scooped?

seems like they use the normal transformer architecture versus deep fold's more specialised machine-learning approaches.

Pssst they'll realise scientists hand out here too

They can keep on doing stuff like this that's open-source and beneficial to society.