The house was built in 1776 by Robert Baldry, who bought the plot of land - Sandpit Piece - for five guineas, from Gerard Vanneck, who was soon to rebuild Heveningham Hall. A stone still to be seen on the chimney stack records the year and Baldry's initials. It was a modest two-up, two-down dwelling, incorporating materials salvaged from other buildings, including a Palladian window first placed perhaps in the front of the house and then, in 1814, installed in an extension at the east end, when a large bay window and a stucco finish brought the front facade into contemporary fashion.

On Robert Baldry's death in 1806, the house was let until 1813 and then sold to Anthony Collett, MA of University College, Oxford, and the Heveningham incumbent. He paid £420 and a further £50 for fixtures. Collett was a man of some means - he owned 600 acres in the area - and could afford the improvements which made the house more substantial and elegant. It was probably stuccoed in accordance with contemporary fashion at this time. William and his wife Ann Rachel, and their children (the youngest of four died in 1821), remained at the Rectory until 1826. The house then passed to the eldest son Anthony but he moved to Bury St Edmunds and the house was leased. The 1841 census shows it occupied by fifty-six years old Simon Smyth, who farmed in the area, his wife Pheobe, and two teenage children. By this time further extensions to the back provided accommodation for two live-in servants. The younger Anthony Collett sold the house for £600 in 1847, to a wealthy local philanthropist, Edmund Holland, who presented it to the Norwich diocese for use as a rectory.

Until then Ubbeston had no "parsonage house". Samuel Badily, vicar for over fifty years until his death in 1854 at the age of 84, had lived in Yoxford. He rode the round trip of nine miles to take services and the Vannecks built him a stable adjoining the churchyard for his horse. The first resident vicar of Ubbeston - from 1856 - was Robert James, then aged thirty-five and with five children, three of them born in the virtual wilderness of Prince Rupert Land where he had been sent by The Church Missionary Society. A year later his wife Emma gave birth to a son whom they christened Rupert. Robert James died in 1876, to be succeeded by Edwin Watkins, another former missionary in the Hudson Bay area, where his children too had been born. The family were either very frugal or had private means since the church accounts for the next thirty years show a succession of gifts from them to church and parish, including a pipe organ to replace Robert James's harmonium, and a beef and plum pudding dinner for every family to celebrate Queen Victoria's golden jubilee.

After his wife died, and his engineer son Arthur moved away, Edwin lived on in the house with his two daughters. Ellen Georgina died in 1898 and her father in 1907. Then aged seventy-eight, he set off on his bicycle to visit parishioners and was found dead at the top of Clay Hill, within sight of the church which he had served for thirty-one years. Alice continued to live in the parish until her death in 1935. She did not live to see the sale of the Rectory in 1939, the start of a process which led to the secularization of the church itself in 1974.

Veronica Baker-Smith, Ubbeston, March 1995