The origins of the hall date to the 11th century. William Peverel held the manor of Haddon in 1087, when the survey which resulted in the Domesday Book was undertaken. Though it was never a castle, the manor of Haddon was protected by a wall after a licence to build one was granted in 1194. The hall was forfeited to the Crown in 1153 and later passed to a tenant of the Peverils, the Avenell family. Sir Richard de Vernon acquired the manor in 1170 after his marriage to Avice Avenell, the daughter of William Avenell. The Vernons built most of the hall, except for the Peveril Tower and part of the Chapel, which preceded them, and the Long Gallery, which was built in the 16th century. Richard's son, Sir William Vernon, was a High Sheriff of Lancashire and Chief Justice of Cheshire. Prominent later family members include Sir Richard Vernon (1390–1451), also a High Sheriff, MP and Speaker of the House of Commons. His son Sir William was Knight-Constable of England and succeeded him as Treasurer of Calais and MP for Derbyshire and Staffordshire; his grandson Sir Henry Vernon KB (1441–1515) Governor and Treasurer to Arthur, Prince of Wales, married Anne Talbot daughter of the Earl of Shrewsbury and rebuilt Haddon Hall.