The ruins of the medieval castle at Pevensey stand in one corner of a Roman fort, on what was once a peninsula surrounded by the sea and salt marshes. The Roman fort, named Anderida, was built in about AD 290. It is one of the largest surviving examples in Britain, with two thirds of its walls and towers still standing to almost their original height. Unlike most Roman forts that were built to a standard rectangular plan, Anderida has an irregular oval layout that follows the shape of the peninsular.

In September 1066, William, Duke of Normandy, landed his invading army at Pevensey, and immediately set up a defensive camp within the walls of the old Roman fort, before marching on to Hastings. Following his victory, William gave Pevensey to his half-brother, Robert, Count of Mortain.

The castle developed gradually over the centuries with phases of building interspersed by periods of decline, including the slighting of the castle by King John in 1216. The castle was rebuilt and remained occupied into the 15th century, by which time it served as a state prison. The castle's strategic location as a possible landing point for foreign invaders gave it an importance that led to several sieges over the course of its history, and even after it had fallen into ruin it was put to use for the defence of the country. A gun emplacement was built at the time of the Spanish Armada, and during the Second World War machine-gun posts and billets for troops were created within the remains of the castle. The pillboxes can still be seen at the castle, camouflaged to look like the rest of the building.

Entrance to the Roman fort is free, but there is a charge to enter the medieval castle.