Below is a map of Rocky Point from Meacham's 1880 Atlas, page 136. I have the Warren Farm house marked in red on this map. I've been told there's an image of this house at the PEI Public Archives.
The following information comes from Parks Canada website - http://www.pc.gc.ca/eng/lhn-nhs/pe/amherst/natcul/natcul4.aspx
When Charlottetown was officially named as the capital of St. John’s Island in 1768, the garrison was permanently removed from Fort Amherst to Fort Edward, across the harbour. The Fort had been in a state of disrepair for many years and was not worth salvaging. The fort was demolished in the 1770’s, and by 1779 there was nothing left of the fort but the ditch.
There were several tenants of the fort lot through the rest of the 18th century, though perhaps none as prominent as the first (and only) Governor of the island, Walter Patterson.
Patterson acquired the land through complicated means in 1773. He and his wife, Hester Warren, built an elegant farmhouse and offices on the lot. They named it Warren farm after Patterson’s wife. Patterson was removed from office in 1786, and the lot was left vacant until 1796, when Patterson’s title to the land was nullified. A variety of landowners farmed the land until 1959, when owner John Hyndman sold the land for its creation as a national historic park. The site was officially opened in 1973.
....About the centre of the burying-ground is recorded the death of Job Bevan, son-in-law of Will. Pepperal. He did duty at the block house as a soldier in the artillery when he came to the Island. The recollections of what it was, if we read them, would throw some light upon the hidden past.
Many of the principal men who figured in our early history, lie here with no stone to mark their graves. We are reminded of them and their acts by members of the family who have been highly honored. A monument to the Hon. Geo. Wright, Surveyor General, who administered the government at five different periods, is a partial record of a family mixed up with all our early public transactions. His father, Thomas Wright, was on the Survey with Capt. Holland in 1765, and was appointed Surveyor General Sept. 1770. He resided with Patterson at Warren Farm, occupying one of those " extensive offices" which the governor erected. As stated before, Mr. Wright was taken prisoner by the Americans and conveyed to Boston, was a supporter of the Governor in his political measures, was one of the commissioners appointed on the suspension of Chief Justice Stewart, and subsequently in 1788 made an assistant judge...
PS. I was searching through Earle's Pictures Restoration's Photos on Facebook yesterday (Mar. 12, 2014) and found this photo posted
Caption: Hoisting up the Hay at the Warren Farm, on the Site of the old French Capitol of Port LaJoie Prince Edward Island Canada.