Looks sweet! What kind of projector are you using? I really want to do this but I have a large tree in one half of my yard...
You are correct that the angles will transfer to your finished product. If you don't angle your overlay ( say stone walls) they will look off. What I did was draw in reference lines of my siding so I could match my images to. Last year I did a decrepit run down house with peeling painted siding. My reference lines were used to line up my peeling painted siding image, so I even had everything lined up plank for plank.
M View attachment 323993
Here is a reference image, albeit sideways.
Edit: I don't remember what software your using for video editing, but with serif, if you hold the control key then grab a corner of the video frame, it makes it easier to adjust for this situation. Other software editing programs can also do this by adjusting your video tracks in "3-d" space, just serif made it super simple. That's the main reason I used serif to begin with, the ability to make quick easy adjustments to match those weird angles.