The oldest part of Kellie Castle is the north-west tower, built in the early 16th century on the foundations of a much earlier structure. The family responsible for building the tower and further developing the castle were the Oliphants, who owned the castle between 1360 and 1613. In 1573 a new tower was completed fifty feet to the east of the original one. In the period between 1573 and 1606 the two towers were linked by a new range that ended at another tower in the south-west, creating the T plan layout that remains today. This final stage of construction proved too expensive for the Oliphant family and they were forced to sell the castle to clear their debts. It was purchased by Sir Thomas Erskine, a favourite of King James VI. In 1617, during his only return visit to Scotland since becoming King of England, James stayed the night at Kellie Castle while travelling through Fife.

Few changes were made to the Castle after the time of the Oliphants, and by 1829 its contents were sold. It was home to a miner for a few years and then used as a barn before being leased, in 1878, to James Lorimer, Professor of Public and International Law at Edinburgh University. The Lorimer family set about restoring the castle and remained resident until 1970 when ownership passed to the National Trust for Scotland.