Old Wardour Castle was built at the end of the 14th century for John, the 5th Lord Lovel. It was built at a time when large, baronial houses were being constructed across the country, with more emphasis on style and comfort than on defence. The hexagonal structure was probably influenced by contemporary French design. The Lovel's supported the Lancastrian cause in the War of the Roses and as a result had their estates confiscated when Edward of York defeated the Lancastrians and became king in 1461.

In 1547 the castle was sold to Sir Thomas Arundell of Lanherne. The castle was confiscated when Sir Thomas was executed for treason in 1552, but was later acquired by his son, Sir Matthew Arundell, in 1570. By this time the house would have seemed old fashioned and uncomfortable by the standards of the day and Sir Matthew set about refurbishing the property. Doorways were redesigned and windows enlarged to let in more light; a new gallery was added and the house was decorated and furnished to a very high standard.

During the Civil War (1642-48) the Arundell's were loyal to the king. In May 1643 the castle was attacked by Sir Edward Hungerford, a Parliamentarian commander, and after a short siege the castle was surrendered to him. In December, Henry Arundell led a Royalist counter-siege. Whether by design or misfortune, gunpowder mines laid in a drainage tunnel beneath the castle produced an explosion powerful enough to cause a large part of the castle to collapse. The damage was deemed as beyond repair and the ruined castle would later become the focal point of the landscaped grounds of the new house built at Wardour.