Castle Campbell is sited in a particularly attractive position at the top of Dollar Glen, a steep sided, wooded ravine. Once known as Castle Gloume, it stands between two streams, the Burn of Sorrow and the Burn of Care. The oldest surviving part of the castle is the tower house built in the mid 15th century by Colin Campbell, the first Earl of Argyll. In 1490 he was granted permission to change the name of the castle to Castle Campbell.

The tower house is in an excellent state of preservation, with four floors, three of which are vaulted. There are fine views from the roof. A more recent range of buildings built in the early 16th century has been a ruin since Royalists burnt it down in 1654, after the castle was garrisoned by Cromwell's troops. The tower house was maintained and used as a residence by the Argylls until the early 19th century.