The practice of stepping the aperture down serves when the atmospheric turbulence renders a large aperture incompatible for the nights viewing.
It has to do with the air cells in the atmosphere that are floating around and the size of them, but not getting to technical , stepping down on such nights can increase the quality of the image although at the expense of the resolution of the full aperture.
So it counteracts itself.
Also removes spider diffraction spikes as well , not a major concern and reduces glare on the moon, plus reduces the coma effect of the main mirror. Fast newtonians need to have the planet kept in the center of the FOV or increasing coma towards the edge reduces contrast.Longer FR's in a newtonian reduce coma and gives a larger FOV free of coma.
This is why some purist planetary observers state that smaller unobstructed refractors of 6-8" are all that is required.
I don't fully agree with this , but looking through an unobstructed larger high quality refractor is some thing to be seen ones self.
A well, designed , dedicated high optical quality Newtonian can compare quite well to a HQ refractor for high resolution work at a fraction of the cost.
I doubt you will get a 5" mask with a 10" Newtonian but assume this is just an example.The hole needs to be between the secondary and the outer circumfrence of the mirror in size.
So 254mm less the secondary size devided by 2 will give the maximum hole size but go a little under this due to turned down edges ect of the main mirror.
With a Newtonian if the native focal ratio of the mirror (not stepping down) is increased a smaller secondary can be used giving a similar fully illuminated field.
This reduction in secondary size
(of diameter) increases contrast.
Most Newtonian's of around f5 -f6 have around 20%ish secondary obstruction .
This is better than say the 30-40% of a SCT.
This really only comes into play on high resolution work for planets and does not effect low power DSO work that much.
Hence true RC scopes of under f8 have obstructions near 50% but is not noticable for DSO's
A Newtonian can have even a smaller secondary of under 20% at the expense of a large fully illuminated field and as such may not fully illuminate a long FL low power eyepiece or CCD chip.
This is not a problem if the scope is a dedicated planetary Newtonian as small FL eyepieces ( say under 12MM) are all that is used and require a smaller F.I.F.V.
As far as to how small you can go with the secondary , any thing under 20% you most probably won't notice the difference , visually any way.
That being said there are commercial and DIY Newtonian's and variants out there with only 10% obstruction ratios , but these are in the order of f6-10 native focal ratios to be able to do this and requires good designing to get the most out of the secondary. ( another story)
My 12.5" Newtonian has a FR of f6 with a 17% obstruction ratio and it's planetary performance is first class and have only on a couple of occasions stepped it down to just over 100mm.
Was not that impressed with the stepping down due to the loss of resolution and prefer to wait for a night of steadier atmosphere.
Hope this confuses you even more
P.S. when people talk about a SCT of f10 don't think that it is easier to make an f10 than an f5 rule applies here.
The main mirror of a SCT has only a primary focal ratio of f1.9-2.1 and the cassegrain secondary a multiplier of about 4-5 and are a hard beast to make extreamly well, even if it is only a spherical surface. QC and tolerances have to be a lot better at f2 .
A link with interesting articles from R Royce .
This guy makes professonal mirrors down to f1 and lenses so kinda knows what he's talking about.
http://www.rfroyce.com/thoughts.htm
Mark