Richmond Castle is dramatically sited at the top of a high cliff overlooking the River Swale. The castle uniquely contains two types of keep, Scolland's Hall, an early French style 11th century keep, and the much taller, 12th century tower keep that still dominates the castle.

The castle never saw military action and was in a fairly ruinous state by the beginning of the 16th century. It received repairs in the 19th century when it became the home of the North York Militia, and was still in use by the military during the First and Second World Wars.