Construction of the magnificent Norman keep at Castle Rising began around 1140. It was built for William D’Albini II, who had recently acquired great wealth and power through his marriage to Alice of Louvain, the widow of King Henry I. The castle features some ornate decoration and was built at great expense using imported stone. The large hall keep had most of its principle rooms, including the great hall, on the first floor. The ground floor was primarily used for storage. A forebuilding containing an impressive staircase leading to a first floor entrance vestibule provided access to the great hall. A large arch with Romanesque decoration served as a grand entrance to the hall. This arch was blocked up and a fireplace installed during the 16th century, and the entrance to the hall is now through a side door and passage.

The keep sits in the middle of an oval bailey surrounded by a large earthwork. The earthworks are so high that very little of the castle can be seen from outside them, but when they where first built they would have been much lower and the castle would have been visible for some distance. The earthworks were heightened at a later date, possibly in the 1170’s. There are some remains of a wall that once stood on top of the earthwork bank. There is a smaller outer bailey, also surrounded by earthworks, and access from this to the inner bailey is via a bridge over the dividing moat and through a small gatehouse.

The D’Albini line failed in 1243 and Castle Rising passed to the Montalt family. They held the castle until 1331 when Edward III granted it to his mother, Queen Isabella. Despite being implicated in the death of her husband, King Edward II, she was not held prisoner and lived a life of considerable luxury. The ruins of a range of buildings to the south of the keep are all that remain of new lodgings she had built for herself. When she died in 1358 the castle passed to Edward the Black Prince. He appears to have been the last owner to spend considerable sums of money on the maintenance of the castle and following his death it gradually fell into decline. In 1544 King Henry VIII granted the castle to the Howard family who have retained it to this day.