>A pretrained model without deliberate alignment is by default aligned to the average person in the developed world, since that's what's inside the pretraining corpus - stuff on the Internet made by humans.

I don't think this is true or a useful way of thinking about it. If the training process makes the model aligned to it's content, then the models are 1. aligned to a random subset of Internet content, weighted by text volume and being easy to scrape, 2. aligned to the training process that makes models chatbots that answer your question instead of just continuing your passage in a similar style. Neither of these are necessarily good enough, IMO.

And that's taken it as a given that the training process can be said to align the models to the authors of the content by default, regardless of what that content actually is. I don't think that should actually be a given.

>You may not personally find the default alignment aesthetically pleasing, but it's humanity. We should set the initial conditions of this new era faithfully, and let it unfold naturally.

Strongly disagree, I think the assumption that natural = good is incorrect and harmful. Polio is natural. And to even call the model's "unaligned" state "natural" seems like an enormous stretch.

You claim the pretraining distribution is not good enough, but provide no alternative methods that are better. The pretrained model is currently our best approximation of humanity's average objective function despite it being a random subset of the Internet. And you're correct that instruction-tuning increases misalignment, so just release the base model.

On "naturalness": you've redefined and strawmanned what I meant by "natural". In the original context, I was referring to the undisturbed unfolding of an era preconditioned on the fact that these models are aligned to humanity's average. That has nothing to do with Polio being a virus found in nature.