I used 3/4" finials which are a tad bit small. I had to use a drill press and vice to open up them slighty to fit (I forget what drill bit size). I've heard of others using heat gun to make them (or the PVC) more pliable. Great idea with the dowels!....
You can also order the finials with a 3/4" base, but I'm not sure if that would fit over the conduit properly in order to eliminate the dowel approach. Perhaps someone else can chime in on the 3/4" base for us. Myself, I'm glad I did it in a way that allows me to remove the finials if I wish without doing any damage.... (think creative changes possible, but you'll see what I have in mind next HW).
In retrospect, the dowel size could be cut in half to about 3". That would equal about an inch inside the finial base and the rest in the conduit. The final thing I would change is not using a hole saw, but an actual 3/4" drill bit, since that would be much faster (I used what I had already on hand).
TIP: I found that I wasted a LOT less spray paint if I painted the pieces before I assembled the fence.