Substantial earthworks and some stone remain from this early Norman castle, built around 1170-80 by William de Warenne, Earl of Surrey and Chief Justiciar to William the Conqueror. Large earth banks surround a bailey, and at one end is a motte with the remains of a keep. This was originally de Warenne’s hall, which was surrounded by a simple ringwork bank. Some time around 1140, during the civil war of the reign of King Stephen, a motte was raised around the reinforced walls of the hall which became the base for a much stronger, although probably never finished, keep. When King Henry II came to the throne the keep was demolished, along with many other unlicensed fortifications that had sprung up across the country during the conflict.

The castle remained in use until the death of the last Warenne earl in 1347. The de Warennes founded a town alongside the castle, surrounded by a defensive earth bank and ditch. The main entrance was a stone gatehouse, now known as the Bailey Gate, which still stands in the village.