Hi,
I am new to this, but are you worrying too much. maybe you are seeing periodic error and with guiding all would vanish

.
Also, my guess is that if its not polar aligned then its just not polar aligned. After you iterate a few times then except for periodic error it should stay exactly on the star you are on.
I tried a test last night. Usually i iterate back and forth a few times and it takes me around an hour to set up each time. I dont have a fixed place to image. The other night i just did a very rough RA align and assumed the DEC was OK from the previous session. I will try this again as in this case i cut down my setup time to only 20 minutes rather than over an hour. Now if i could only focus properly
With guiding using PHD i could see no drift using 8 minute subs with this rough align, this is about as long as i wish to go with a DSLR. Admittedly my focal length is only 600 mm, but hey it works for my setup
By rough i mean only minimal vertical movement after about 3 minutes, and also by rough i mean i didnt do an iteration with a barlowed reticle.
Paul