This looks like a good pre-read for Nathan Lambert's https://rlhfbook.com/