The fairly conventional appearance of Huntingtower Castle hides the fact that until the 17th century it consisted of two separate tower houses built just a few metres apart. The earliest part is the eastern tower which was originally a gatehouse but was converted around 1500 into a three-storey residential tower house. The first floor hall has a tempera-painted ceiling dating from about 1540, which was hidden behind later panelling and was only revealed during restoration work in 1913. There are also traces of painted plasterwork which date from an even earlier period.

The western tower was built soon after the eastern tower. It is also a three-storey tower house, although it is larger then the eastern tower and has a four-storey chamber block at the southwest corner. It has been speculated that the unusual arrangement of the two separate tower houses might date from 1480 when the estate was divided between two sons of the first Lord Ruthven.

Until 1600 the castle was known as the House of Ruthven and was the home of the Ruthven family. In 1581, the young King James VI created a new earldom for Lord Ruthven who became Earl of Gowrie. The following year an infamous event took place that became known as the Ruthven Raid. While King James VI was in Perth returning from a hunting trip, he was persuaded to visit the House of Ruthven. Gowrie and his Protestant allies were concerned about the influence on the king of his chief advisors, The Duke of Lennox and the Earl of Arran, both of whom were Catholic sympathisers. They held the King captive in the House of Ruthven, for ten months, during which time Gowrie and his fellow conspirators effectively ruled the nation. Following his escape the King initially pardoned the conspirators, but in May 1584 Gowrie was beheaded at Stirling. The House of Ruthven was inherited by his sons, but they were killed in 1600 during a failed attempt to once more capture the King. In revenge the King had the name of Ruthven abolished, their arms were cancelled and deleted from the Book of Arms and all their property was forfeited and taken by the Crown. The name of the House of Ruthven was changed to Huntingtower Castle.

The castle remained Crown property until 1663 when James, fourth Earl of Tullibardine, became the new owner. Following his death in 1670 the castle passed to John, Second Earl of Atholl. In the late 17th century the gap between the two towers was filled by a new building that finally connected all the accommodation in a more conventional manner. The castle remained home to the Atholl family until 1805 when it was sold and used as accommodation for the workers of a nearby cloth-printing factory.