Aberdour Castle consists of a range of buildings that date from various periods, the earliest of which are now the most ruined. It started as a hall house, probably built around 1200 by Alan Mortimer. In 1325 King Robert the Bruce granted the lands to his nephew, Thomas Randolph, Earl of Moray. In 1342 the property was granted to Sir William Douglas and it has remained in the Douglas family ever since.

In the 15th century the hall house was heightened to become a tower house. In the mid 16th century a central range was added to the south and east of the tower house to provide better accommodation for James Douglas, 4th earl of Morton. James was Lord High Chancellor from 1562 to 1572 and ruled Scotland as regent from 1572 to 1578 after the abdication of Mary Queen of Scots in favour of her infant son, James VI.

The last major addition to the castle was the east range built for William Douglas, 7th earl of Mortimer. He was Lord High Treasurer of Scotland between 1630 and 1636, and was one of the most powerful and wealthy men in the country. Most of the first floor of the new range was a long gallery where Sir William could display his collection of paintings and entertain his guests.

In the late 17th century the castle was badly damaged by fire. Only the east range was repaired, but plans to completely demolish the tower house and central range were abandoned due to the cost. Much of what was left has collapsed over the years. In 1725 the earl and his family moved to nearby Aberdour House. The east range continued in use as barracks, school and masonic lodge before being placed in state care in 1924.