Llawhaden Castle was built by the Bishops of St Davids cathedral. It was probably Bernard, the first Norman bishop, who founded the castle in the early 12th century to protect the surrounding lands. The ditch from this early ringwork is still visible today. The castle was captured and destroyed by Lord Rhys in 1192, but it was soon rebuilt and during the 13th century a towered stone curtain wall was constructed. Towards the end of the century Bishop Thomas Bek upgraded the residential accommodation at the castle and established a market town outside its gates.

In the late 14th century bishop Adam de Houghton began a major rebuilding scheme. He added a high stone curtain wall with multiangular towers around the southern half of the castle. A lavish residential range was built inside the walls that would have provided comfortable accommodation for the Bishop's guests. The entrance to the castle was provided with an impressive twin towered gatehouse that still rises to its full height, although it is now only a facade, the inner part having collapsed or been demolished.

At the time of the Reformation in the 16th century, Bishop Barlow moved the principal residence to Abergwili near Carmarthen. Llawhaden Castle was stripped of saleable materials, and in 1616 Bishop Milbourne was granted a licence to demolish it.