Sir Roger Fiennes was granted a licence to build Herstmonceux in 1441. The castle was one of the earliest to be built of brick and was constructed in a lake so that the water acted as a large moat. The castle was well appointed, designed more for comfort than for defence. Unfortunately the castle was not well maintained by some of its later owners. By the end of the 18th century much of it had been demolished to provide building materials for Herstmonceux Place, further up the hill.

During the first half of the 20th century the ruins were transformed into an impressive home and the external appearance of the castle was restored. In 1946 the castle became the home of the Royal Greenwich Observatory. They remained for the next forty years before moving to a new home, and the old Equatorial Telescope Buildings are now open to the public as a 'hands on' science centre.

The grounds and gardens are open daily throughout the summer but the castle is now a working study centre and is not open to the public; however, guided tours are conducted Sunday to Friday, subject to availability. Please phone for confirmation of tour times.