Not much remains of this once large castle, and not all the ruins are genuine, the tower on top of the motte is a mid 18th century folly. The original Norman castle was built in 1071 for Hugh de Avranches but it was almost immediately transferred to Henry, Lord of Ferriers and Chambrais in Normandy.

In 1174, William Ferrers came into conflict with the crown causing Henry II to lay siege to the castle, and subsequently to order that it be demolished. The castle was rebuilt, but in 1263, Prince Edward (the future King Edward I) also attacked the castle, again causing great damage. In 1265 Henry III gave Tutbury Castle to his younger son Edmund, created Earl of Lancaster in 1267. It has remained in the hands of the Earls and Dukes of Lancaster ever since.

In 1362, John of Gaunt, second Duke of Lancaster, gained Royal permission to repair the castle, and over the next century new walls, towers and buildings were added. However the castle was already in a poor state of repair when Mary Queen of Scots was imprisoned at Tutbury in the late 16th century. In 1646, during the Civil War, the castle fell to Parliamentary forces after a 3 week siege and was ordered to be destroyed, leaving the ruins visible today.