Norham Castle was founded by Bishop Ranulph Flambard of Durham as the administrative centre for his most northern territory. Built in 1121 at a strategic crossing point on the River Tweed, it was soon captured by the Scots in 1136, and although returned to the bishop shortly after, it was again besieged and largely destroyed by the Scots two years later.

Due to its strategic importance, Henry II ordered that the castle be rebuilt in stone. The current bishop, Hugh de Puiset, built extensively between 1157 and 1170, including the construction of the Great Tower, which had two storeys and a basement. Norham's location on the border with Scotland led several armies to its walls. The castle successfully withstood sieges of forty days in 1215, almost a year in 1318 and seven months in 1319. Norham was finally captured in 1327, but was restored to the bishop the following year.

During the 15th century further building work was carried out, including the addition of another two floors to the Great Tower. In 1497 the Scots again besieged the castle unsuccessfully, but they did great damage with the enormous cannon known as Mons Meg, which can now be seen at Edinburgh Castle. In 1523 the castle finally did fall to Scottish cannon, only to be returned three weeks later when the Scots were defeated at the Battle of Flodden. By the end of the 16th century the castle was in a very poor state of repair, and when, in 1603, James VI of Scotland also became James I of England, Norham's position on the border with Scotland lost its strategic importance and the castle was left to fall into ruin.