One of the largest castle ruins in the South West is sited on a long, thin shale outcrop in a wooded valley on the northern edge of Dartmoor. The castle was probably built after 1068, and is mentioned in the Domesday Book (1086) as the centre of the estates of Baldwin de Brionne, Sheriff of Devon.

The castle started out as a motte, which gained a stone keep in the late 11th century. A second building was added to the keep in the early 14th century, along with a range of new buildings further down the ridge. The castle was abandoned in 1539 after its owner, Henry, Marquis of Exeter, was found guilty of conspiracy and executed by Henry VIII.