Westbury House was the principal residence for John S. "Jay" Phipps, his wife Margarita "Dita' Grace, and their four children. After their parents' deaths in the late 1950s, Jay and Dita's children decided to open their family estate to the public in 1959.
The exterior and interior of Westbury House as well as the basic landscape were the creation of self-taught English designer George Crawley (1864-1926). The brick and limestone exterior echoes English country houses of the late 17th century. The architectural ornamentation of the interiors and selection of furniture reflect English styles of the early to mid-18th century.
Prior to Jay's commission for Westbury House, Crawley was known to the Phipps family as he had worked on interior designs for the residences of Jay's father, Henry, and his sister Amy Guest. Before his marriage Jay had purchased land in Westbury and intended to use it for hunting and polo. But shortly after his marriage he decided to settle in New York and use this land to build a residence. Design began in late 1904 and construction in 1905. Since Crawley did not have formal architectural training, Phipps hired the up-and-coming architect Grosvenor Atterbury to oversee the building's technical requirements.
Major construction was completed in 1906 and the family took up residence in spring 1907. The most significant modification of the residence was in 1911 when a service wing was added. This project relocated the food preparation areas that were adjacent to the dining room (now the Study) as well as the servants' living quarters, which were on the third floor. At about the same time an extension was added in the space that would eventually become the Dining Room.
In the 1920s two smaller revisions occurred: one of the two drawing rooms would become enlarged and transformed into the Music Room and the West Porch was expanded and enclosed.