What do you find hard about it? Is it a conceptual thing or the tools themselves? I'm very far from a sysadmin, but it's just such a ground level using-linux type thing in my brain, I don't think I ever thought I had a choice about learning it!

Owner/Group/World rwx permissions are ground-level. I'm not sure I'd put ACLs, or SELinux, or AppArmor in the same sphere. Those start of at "arcane" and then get more complicated, at least that's my impression, and I've used linux for decades.