Dryslwyn Castle stands on top of a hill overlooking the Tywi valley. Its date of construction is unknown but the similarity between it and neighbouring Dinefwr Castle suggest that it was built at a similar time and possibly by the same person. The most likely builder was Rhys Gryg who occupied Dinefwr in the early 13th century, or possibly his son Maredudd, who inherited Dryslwyn from his father.

By the late 13th century the castle at Dryslwyn had developed into the largest native Welsh castle in South Wales. In 1277 the English king, Edward I sent an army into Wales to defeat Llywelyn ap Gruffudd. Maredudd's son, Rhys, who had inherited Dryslwyn after his fathers death in 1271, surrendered without a fight and was allowed to keep his castle. Dinefwr Castle was not as quick to surrender and as a result was forfeited by the king. Dryslwyn now had an English neighbour, a situation that was not well received by Rhys who felt he had a claim to the lands. In 1287 Rhys, enraged by years of border disputes with his English neighbours, captured the castles of Dinefwr, Carreg Cennen and Llandovery. The English response was swift and an army of 11,000 men recaptured the castles and defeated Rhys after a three week siege at Dryslwyn. Rhys escaped but was eventually captured and executed for treason.

Following the battle the English repaired the castle, although there was very little new building after this time. In 1317 the castle was given to the unpopular Hugh Despenser, the king's favourite. Dryslwyn was attacked and damaged during a Marcher Lord revolt against Despenser, but was subsequently repaired. In 1403 the Welsh constable of the castle declared for Owain Glyn Dwr, who used the castle as a base from which to mount attacks on the surrounding area. When he withdrew from the castle the English moved back in and deliberately dismantled the fortress to prevent any future threat.