Remix.run Logo
▲ octoberfranklin 2 hours ago

Curious: how can there be a correct prediction for the next element of a "shuffled" protein? The shuffled proteins don't occur in real life, so what makes one answer correct and another incorrect?

▲anonymous_sorry 2 hours ago | parent [-]

It is a rather odd control. Is it just to confirm that Claude isn't reading the sequence off disk?

▲gilleain an hour ago | parent [-]

Perhaps. They explain why:

> Shuffling keeps the letters and destroys their order, so the difference in accuracy measures what the model knows about the sequence itself.

So by analogy, if you know a word in English starts with 'Q', you could reasonably predict the next letter would be 'u'. By shuffling, you might encounter the letter Q later on, and not successfully predict a 'u'.

Put another way, the shuffled sequences can only use frequency analysis for prediction, while the unshuffled ones have local and global order signals.

▲octoberfranklin an hour ago | parent [-]

Yes but what is the source of truth for the next element of a shuffled sequence?

For a natural sequence, the observed molecule in nature is the source of truth.

▲gilleain an hour ago | parent [-]

We are definitely going beyond the borders of my understanding of sequences here, but I would guess the source of truth is just the expected frequency distribution of amino acids? So 'G' (glycine) is most frequent, and something like 'W' (tryptophan) is least frequent (dependent on organism?).

However I thought the point was that the _difference_ between the shuffled prediction and the unshuffled is the measure here. So shuffled is the baseline, and the gap above that is how good the prediction is.