Scarborough Castle is situated on a roughly triangular headland that rises 300ft above the sea. There is evidence of Iron Age settlements and the remains of a Roman signal station on the site, but the first medieval castle was built by William le Gros, Count of Aumale, in the 1130's. He constructed a wall on the landward side of the promontory and built a tower where the keep now stands. When Henry II came to the throne in 1154 he demanded the return of all royal castles, and Scarborough, which was built on a royal manor, was one of the castles reclaimed by the Crown.

Henry II spent a lot of money improving Scarborough Castle, adding a large square keep on the site of the previous tower and building a wall to protect the inner bailey. More extensive work was carried out by King John, who improved and extended the castle walls and built a new hall in the inner bailey and a new royal chamber block and great hall in the outer bailey. In 1243 Henry III constructed a new barbican and added a double drawbridge tower - the drawbridges have since been replaced by stone arches.

In 1312 the castle came under a brief siege when Edward II's favourite, Piers Gaveston took refuge there. A shortage of supplies forced him to surrender within a fortnight. Despite promises of safe conduct, his journey back south was brought to an abrupt end when he was captured and beheaded by the Earl of Warwick. Greater resistance was offered by the Royalist defenders of the castle in 1645 when they were besieged by a Parliamentary force. They managed to withstand a five month siege, despite an artillery bombardment that had caused half the keep to collapse within just three days. They were finally forced to surrender on July 25 1645. After the Civil War the castle was used as a prison, and in the mid 18th century a barracks block was added that remained in use until the middle of the next century. The barracks were severely damaged during the First World War when two German warships fired more than 500 shells at Scarborough on the morning of 16 December 1914. As a result of the damage the barracks were demolished.