Castle Fraser was mostly built between 1575 and 1635. It is a large z-plan tower house with two projecting wings that form a courtyard. Michael Fraser began the construction of the castle, but he died in 1588 leaving the castle unfinished. The work was continued by his son, Andrew Fraser, who in 1633 became the first Lord Fraser.

The oldest part of the castle is a rectangular tower house which was built in the 15th century. Michael Fraser kept this building and added two new towers on diagonally opposite corners, forming the distinctive z-plan shape. The tower on the north-west corner is square while the tower on the south-east corner is round and slightly taller than the rest of the castle. It has a flat roof which can be accessed by visitors giving good views of the surrounding countryside. Finally the east and west wings, or ‘laigh biggins’, containing the kitchen, bakehouse, brewhouse and servants quarters were added.

The castle was home to three different families during its history, two branches of the Fraser family and finally the Cowdray family who carried out a restoration of the castle after they bought it in 1921.