During the Wars of Religion, the castle was entrusted to Geoffroy de Vivans, born in Castelnaud. Nicknamed the Battler, he was feared throughout the Perigord and his reputation alone kept attackers away from the castle during that period.
In the 15th century, new living apartments were set out to improve the comfort of the medieval fortress. Despite all that, the Caumont family left Castelnaud to go live in their new castle in Les Milandes.
In the 17th century, the castle was rarely occupied. Abandoned during the French Revolution, it continued to deteriorate and even served as a stone quarry in the 19th century.
Listed as a Historic Building in 1966, the castle has subsequently undergone spectacular restoration projects and, since 1985, it houses the Museum of Medieval Warfare.