UPDATE
We took a trip to the registry of deeds in Cambridge. Very interesting time. Basically you start with your deed (ours was in book 52,882) and just start working backwards. Before we went we had done a lot of online research - in addition to the map and genealogy research we did above that gave us two potential owners - A. Johnson Jr in 1831 and P. Fitts in 1870 - I went through the online records of the registry and found the owners back to 1945 or so (book 6,xxx ).
I'm not kidding they really do have 60 odd thousand books. In racks. one by one.
Anyway, we get there and since our starting point was 1945 we got sent right to the basement where the "old books" are. Surprisingly were were able to quite quickly get into the 1800s. The house was owned by 3 sisters in the 30's who inherited it from there father Patcrick Fitspatrick, and the deed stated he was also known as "P. Fitts" !! and referred to a will and an old mortgage. That gets us to an 1858 mortgage from Edward Blood.

Then the trail ends. At that point the deeds no longer refer to the prior seller so we have to go back and forth from the basement up to the 4th floor to look up names in the grantor/grantee index books. We were having a hard time finding whomever sold it to Blood so we skipped ahead and looked up Abner Johnson. There are LOTS of sales involving Johnson, all for large tracts of farmland - 46 acres here, 28 acres, 5 acres, and one for 3 tracts totaling 65 acres that were granted to the seller in a court judgement from someone else, that one said to go read the "sheriffs deed" for a description. I
think that all these are Abner Sr. buying farmland - so one guess is that Abner Jr. just built the house on part of his fathers farmland than it was inherited or sold at his death. Hard to say as all these pots just state some land "and all the building thereupon"
What is fun is that using the 1831 map I can find the plots because they all refer to the road the meetinghouse is on and measurements from neighbors houses and I can find those names labelled on the map. Downside is so far none of the plots listed is in the area of my house- but there is that vague 65 acre plot, I haven't been able to locate it.
In the fall we are thinking of going to the town historical society library to look up more old maps and then take another try at the deeds.
BTW, Starting with our deed in book 52,882 we got all the way back to a deed for Abner Sr. in book 161 ! ( Book one starts at 1648).