The land with various properties that have been built then replaced has been recorded as far back as the 15th century, starting with the Beauchampe family, though there is not a direct reference to this land until the Forster family obtain it through marriage in the 1600's.  This is when the land and the first building is refered to as Haversham Place. 

Current structure probably built in early 1850s, when it was known as Hanslope House. Became "The vicarage" in 1890s.
At various times, some of the outbuildings were also used as accommodation: for grooms, or let to tenants.