FWIW all that is barely different from German (I'm a German native speaker). These implied groups seem like a nightmare if you have to learn how they work. Since I "just know", I've never had to think about it much.

Native English speakers, please correct me if I'm wrong about the "the castle" example.