Robert Ray (also known as Mr. Kay), a distant relative of the caricaturist, was born near Penicuik in 1740 and began life as a carpenter before becoming a successful builder and architect in Edinburgh. Although it was believed he gained wealth through his wife, Janet Skirving, she was not rich when they married, and any property she later inherited came with unpaid debts he had already advanced. Much of his fortune was made through profitable building ventures in South Bridge Street, after which he retired to a house and garden he built at Duddingston, living quietly with a small circle of friends. His wife died suddenly in 1813, and though her niece Mary Hardie was entitled to half their shared property, she made no claim until after his death in 1818, when it emerged he had left most of his estate to trustees for charities. After legal action, Mrs. Hardie won her full share and received over £1700, while the remaining property still allowed the trustees to support Edinburgh charities, including funding the Edinburgh Sessional School in Market Street.