The first castle at Raglan was a Norman motte and bailey which survived until the early 15th century when it came into the hands of Sir William ap Thomas, a Welsh knight who had fought at Agincourt (1415). About 1435 he began building the Great Tower, an unusual hexagonal keep, surrounded by its own moat. The yellow stone from which it was built gave it its name, the Yellow Tower of Gwent.

Sir William's son, William Herbert, Earl of Pembroke, carried on the work and added a great gatehouse and lavish accommodation. The last major period of building was carried out by William Somerset, Earl of Worcester (1548-89), who was responsible for improvements to the Great Hall.

During the Civil War the castle was held for the king, and in June 1646 came under attack from the Parliamentary forces led by Sir Thomas Fairfax. After suffering heavy bombardment for several weeks, in one of the longest sieges of the war, the castle finally surrendered. The castle was slighted by the victorious Parliamentarians, and after considerable effort they managed to topple two sides of the Great Tower. Further damage was caused when the Duke of Beaufort ransacked the castle for fittings for his new home at Badminton, leaving Raglan a derelict ruin.