Yeah, I play with my friend and we had good luck with the castle placement. First castle was of course in the first zone, and when we proceeded into the second, we settled in basically the very center of the map, a bit toward the top. That turned out to be a fine location, as on horseback, nothing was too far. Castle size as well, initially I built is to just fit the existing infra, but when I sensed that there are going to be additions, I built it a bit larger than it needed to be, and now we can fit everything, especially with the little in-house teleports.

I totally feel your way of beating the game on normal, and the bosses on easy. I did that with Persona 5 at points, and Unicorn Overlord as well. I really appreciate that games provide this option.