The lands (which originally included much of the peninsula of Hoxa and other farms on the island) surrounding the present Roeberry House had been farmed by the Gray family since the early 17th century. Thomas Gray, an Udaller, farmed Hoxa and Cletts and called his home (then situated nearer the shoreline), "Roeberry", an old Norse name meaning red rock and headland. His son, William, was granted a charter in 1649 for these lands in Hoxa by the Earl of Morton and they were continually farmed by the Gray family until 1922, when the farm was sold.

Roeberry House itself was built in 1861 and remained in the family until the beginning of the 21th Century. During the Second World War the house was occupied by the Army and used as staff accommodation and administrative offices for a military centre. Temporary huts were built within the walls which had been erected in the late 19th century by Yorkshire builders using stones from a local quarry and remain shielding 4 acres of garden and woodland to the present day.