Until the late 16th century, most of the low lying ground between Rye and Winchelsea was a shallow harbour, called the Camber, protected from the sea by a long series of shingle banks. Between 1512 and 1514 Sir Edward Guldeford built a circular tower at the end of one of these shingle spits to defend the harbour. In 1538 the threat of invasion from Catholic France and Spain led Henry VIII to build a chain of artillery forts along the south coast to protect vulnerable and strategic areas. The existing tower at Camber was incorporated into a new fort built between 1539 and 1544. However the castle was to have a very short active life. By the end of the 16th century the silting of the Camber made the castle largely obsolete and in 1637 the garrison was disbanded. This early abandonment of the castle has meant that unlike other Henrician forts, such as Walmer Castle, that have been greatly modified over the centuries, at Camber Castle the original design still remains.

A visit to Camber Castle today involves a pleasant one mile walk across flat fields. The castle is managed by Rye Harbour Nature Reserve in co-operation with English Heritage and guided walks around the nature reserve and castle are organised regularly. The exterior of the castle is free to visit at any reasonable time, and the interior can be visited on weekend afternoons in the summer.