A Norman castle of earth and timber was built at Skipton by Robert de Romille, soon after 1090. The defences were later rebuilt in stone. The castle is most closely associated with the Clifford family who were granted the property by Edward II in 1310. Robert Clifford immediately began to strengthen the fortifications but he was killed at the Battle of Bannockburn in 1314 leaving the work to be continued by his son Roger.

The castle is strongly situated on a cliff above the Eller Beck. A twin towered gatehouse remains from the outer defences and within these is an inner curtain wall with six towers positioned close to each other that form a D shape enclosure facing away from the cliff. The space formed by the inner walls contains early Tudor domestic buildings around a small courtyard, the Conduit Court, at the centre of which is a yew tree planted by Lady Anne Clifford in 1659 to mark the completion of restoration work after the Civil War. To the side of this enclosure is a Tudor range added in 1535 by the Earl of Cumberland - this is not open to the public.

During the Civil War the castle withstood a three-year siege by Parliamentary forces and surrendered with honour in 1645. Unfortunately the Royalists returned to re-occupy the castle in 1648, and following their removal the castle was partially demolished to prevent any further military use. However the castle did not remain ruined for long. In the 1650's Lady Anne Clifford began the restoration of the castle, although she was not allowed to return the castle to its previous strength and had to make the walls thinner and the ceiling too weak to support cannon. In some of the towers you can clearly see the much thinner walls of the restoration work on top of the original thick walls.