Some more clues. The town historical society gave a talk on the old houses in town last night. It was a long slideshow and the town historian narrated the history of many of the more prominent houses that have been extensively researched.
Our house, being very small, was not in the group of well researched, but I did find one interesting photo of it that gave some clues. It turns out that in the 70s or 80s a group of architecture students did a housing stock survey in town. This is partially where all our circa sign dates came from (and many are wildly off). My house is in the set of photos under the category "colonial houses" (they also surveyed Federals, Greek Revivals, Victorians, etc).

Whats most interesting is the note that a previous town historian had studied the place. I will have to go dig through the society records and see what I can find. I know that the note calling it "Fitz house" is not 100% correct because I have the deed when he bought it in the 1850s and we know its older than that by the construction.
I did learn a lot about building trends in town. The society has lots of well documented cases of houses that were built with salvaged materials of older houses, barns etc that were town down. That could explain why we have floorboards that seem too wide/old for the 1800's with nails that are too modern for the 1700s. Also many documented cases of houses being moved all over the place (across town or just a few feet to accommodate road widening) and many houses burning and being rebuilt on the same foundation. There was a cool story of one house that was moved from the top of a hill all the way down to the center of town with log rollers and an ox team, and the sills broke as it was being pulled across the main street so it ended up sitting in the middle of a road for a month or two while they repaired it!