The earliest part of Edzell Castle is a tower house and courtyard built in the early 16th century. This replaced an earlier motte castle that was a few hundred metres away (the mound is still visible). The castle was home to the Lindsay family, who had inherited the estate through marriage in 1358.

Around 1553 a new block was built along the west side of the courtyard. This contained a new entrance, a kitchen and storerooms at ground level, and private rooms above. Towards the end of the 16th century Sir David Lindsay added another range of buildings in the northwest corner of the courtyard. This included a large kitchen in the basement with a hall on the floor above, and private accommodation in a round tower projecting from the northwest corner. Sir David's most notable contribution is the walled garden built to the south of the courtyard. This is considered to be one of the most remarkable castle gardens in Britain. The formal planting dates from a restoration in the 1930's, but the highly decorative walls were built in 1604. The walls were designed to allow flowers to be planted in niches to reproduce the heraldic devices of the Lindsay family. In between these heraldic displays are carved panels that represent the Planetary Deities, Liberal Arts and Cardinal Virtues. A summer house was built at one corner of the garden with a bath house on the opposite corner.

The castle was occupied by Cromwell's soldiers for a month in 1651, and by Argyll Highlanders during the Jacobite Rising of 1745. The last Lindsay laird had been forced, through debt, to sell the castle to the Earl of Panmure in 1715. Following the Jacobite Rising in which he took part, Lord Panmure's estates were forfeited and came into the possession of the York Buildings Company. When the company became bankrupt in 1764, Edzell Castle was gutted for its building materials and sold on behalf of the creditors.