I mean no, it's not.

Castle Sween was built in the 11th century (and might still not be the oldest).

Dalhousie Castle was built in 1247 and the oldest bit of the surviving castle is from the 15th century. Most of it is from the 17th century. The roof is post-2004.

The actual lesson is, if you want to leave a legacy, hope someone rebuilds it when it falls apart after you're gone.