Hailes Castle stands in a picturesque setting beside the River Tyne. It is one of the oldest stone castles surviving in Scotland, probably built by Hugo de Gourlay in the late 13th century.

The Gourlays lost their land in the early 14th century and Sir Adam de Hepburn became the new owner. The Hepburns made extensive alterations to the castle increasing both its defensive capability and domestic facilities. The castle remained with the Hepburns until 1567 when James Hepburn, 4th Earl of Bothwell and 3rd husband of Mary Queen of Scots, forfeited his lands and fled to Norway. The castle passed first to the Stewarts, then to the Setons and finally, in 1700, to the Dalrymples of Hailes. However it was soon abandoned by the Dalrymples in favour of a new mansion.

Both the original Gourlay tower and a later Hepburn tower have pit-prisons at their base. The older tower was eventually converted into a dovecot. The Hepburns also built a curtain wall around the landward side and later added a range connecting the two towers. The remains of kitchens can be seen in the vaulted basement of this building.