Brough Castle was built in the 1090's in one corner of the remains of a Roman fort. It was one of the first stone castles to be built in Britain, and some of the walls show the herringbone pattern typical of Norman masonry. In 1174 Brough was attacked by the Scottish King, William the lion, and left in a ruinous state after he forced a surrender by setting fire to the castle. In the 1180's the castle was rebuilt by Theobald de Valoignes who constructed a new four-storeyed keep on the site of the previously destroyed one.

Along with several other castles in the area, Brough passed to the Clifford family. In the early 14th century Robert Clifford began to enlarge and improve the castle. The round tower, known as Clifford's Tower, dates from this period. Successive generations of the family continued to improve the castle until abandoning it in 1521 after a major fire. It lay empty for 140 years but was rescued in 1659 by Lady Anne Clifford who restored Brough and the other Clifford castles. However, following Lady Anne's death, her successors, the Earls of Thanet, undid most of her good work by demolishing much of the castle to provide stone for the construction of a new house at Appleby Castle.