I would take "Our town has the castle" to imply that there's a group of towns that collectively only have one castle, and your town is the one that has it.

Compare: "Sacramento is the capital." You could say Sacramento is a capital if the reference group includes Albany, Austin, Tallahassee, etc. Or you could say Sacramento is the capital if the reference group includes Los Angeles, San Diego, San Francisco, etc.