Why is speech a specific, precise red line? Yelling "fire" in a crowded theatre (and I'm aware that the case which coined this phrase was completely stupid, but the phrase itself stands) should probably not be legal, even though it is speech.

Why not? If shouting “fire” causes no harm, what’s the issue? And if it causes harm, isn’t that adequately served by civil lawsuits? Not everything on earth needs to be a criminal matter.