I think it's interesting how even inside tech, so many people seem to view neural network weights as "black magic" utterly beyond human comprehension. I guess that contributes to the sense of mystery around LLMs...

(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)

My main takeaway is that HN had no idea the field of interpretability existed until now

I don't know why because Anthropic has had a lot of blog posts about internal work on it