The ruins of Edlingham Castle stand next to Edlingham Burn in a rugged but attractive valley. John de Edlingham built a large two-storey Hall House in a moated enclosure in the mid 13th century. In 1296 the property was taken over by Sir William de Felton who added a palisade inside the moat and a gatehouse on the north side. In about 1340 his son further improved the castle and added the solar tower, a gate tower and stone curtain walls. The tower contained three well-appointed rooms and was linked to the hall house by a small forebuilding.

After 1420 ownership passed to the Hastings family until 1519 and then to the Swinburnes. These two families did not need the defensive capabilities of a castle, and until its final abandonment in 1650 the property was used as a farm with several of the buildings converted to house livestock.