The Forest of Barden was a hunting forest for the owners of Skipton Castle. In the late 15th century, Henry Clifford, the 'Shepherd Lord', rebuilt one of the hunting lodges and made it his principal residence. During the Wars of the Roses he spent his youth in the Cumbrian fells tending sheep, while hiding from the Yorkists who had killed his father and grandfather. He was restored to his estates by Henry VII, but chose to live in the more rural setting at Barden Tower.

In 1658, Lady Anne Clifford restored the decaying property. Following her death it was taken over by the Earls of Cork, but fell into decline in the late 18th century. It is now just a ruined shell and is only viewable from the exterior.