the vingetting is more than likely coming from the secondary being too small On the standard skywatcher 254mm dobsonian the fully illuminated circle came in a a whopping 6mm..... at 78mm for the 12" F4 like i said you would be looking a minimum of 90 odd mm to start getting close, but more like 100mm to do the job properly!.
Have a look at this
image Just to give you a idea thats close to 400mm from the OTA surface to the sensor + about 150mm to the secondary! It missed out on focus with a Starlight adaptive optics unit + Starlight OAG + QHY9 and filter wheel! by about 5mm.
Im only talking with experience on my own rig but i have done alot of reading and looking at the designs for it. I have even designed a 8" F4 GSO mirror newt and that required a 70mm secondary to get a 20mm 100% illuminated circle! which doesn't even cover my qhy9 properly.
Brendan