Data Rejection Procedures

[gregbradley]

I thought I would start a thread where we can discuss different techniques for data rejection. This will help anyones image processing and its an area that does not seem to be well written up.

I usually only use hot and cold pixel rejection in CCDstack.

It used to be one item in CCDstack 1 but in CCDStack 2 it is 2 separate items. Also hot pixel rejection in CCDstack 2 seems to bog as the slider colours in and then it does nothing apparently for quite a while then it completes its task, so watch for that.

Once its done its selection I then use interpolate the rejected data to correct it.

I have found in experimenting that if I did not do hot/cold pixel rejection I would tend to get little black spots or coloured spots in the background of the image that required the healing tool to correct. Doing this I now rarely have to correct spots in the background.

However the other data rejection tools seem to damage the image when I have used them or select way too much of the image. Obviously the default settings are not right and have to be adjusted. To what I don't know.

I don't see they are of much use to my setup where the images are usually defect free except for satellites etc.

I use median combine. I notice John Gleason said to use sigma reject combine. I don't think that is an option in CCDstack. He must be using another program.

Greg.

[Paul Haese]

Greg, just after you normalize a stack of images, you go into data reject and sigma reject is in there. I do this just prior to final combine in CCDstack. I rarely use the option on a stacked image. I found that it can cause more damage than now.

When you use sigma reject you can vary the percentage of strength. Sometimes cosmic ray strikes will not go away without a really high setting. Odd really because they are so random.

[multiweb]

Here's my flow in CCD Stack.

1_ calibrate (flats/darks/bias)
2_ reject bad pixel map. (interpolate rejected pixels)
3_ star registration
4_ Normalisation
5_ STD Sigma reject. I aim for top image 2% or explicitly from 1.9 when it's super noisy up to 2.5 maximum factor.
6_ bit of cropping then deconv if needed 30 iterations positive constraint
7_ Save as a scaled 16bit TIFF then off to PS.

[gregbradley]

I'll try using Sigma Reject in data rejection and see if it helps. Are you mainly using it to get rid of satellites and cosmic rays?





Looks like Sigma reject is worth trying out. The other ones seem to select too much with their default settings.

Gotta watch saving as scaled as it can black clip. I save as 16bit TIFFs and use curves and levels in PS to bring up the image.

I sometimes use decon and other times not. It can make an image look harsh especially the stars but if done mildly or multiple stengths in luminance layered in PS with appropriate opacities it can be useful.

Its also useful if one colour sub stars are fatter than the others for some reason to make them match the others better.

Greg.

[Paul Haese]

Greg it mostly gets rid of everything but sometimes it just needs a bit more percentage.

[gregbradley]

Thanks for the tip Paul. I'll have to check that out. It would have useful on my recent NGC3046 image. That had more satellite trails in it than any other image I have ever taken. Some subs had 4 or more satellite trails in them. Its busy up there!

Greg.

[strongmanmike]

Wait till you start using your Starlightxpress Greg...all you have to do is dither your lights and then use a median combine and that's it :thumbsup: If your image train is clean, the chip size will likely mean you won't need flats and thus flat darks either :thumbsup::thumbsup: Everything you add to an image ie darks, flats, flat darks will inevitably add some noise along the way so not having to use'em in the first place is the key :thumbsup:

Mike

[gregbradley]

Sounds very good Mike. My camera is on its way to me now.

What do you use to dither? I don't think I can do that with CCDsoft. Although once there was a plugin that allowed you to dither. I should try that again.

Greg.

[Shiraz]

good thread Greg.

Like Mike, I do not use either darks or flats with the 694. Dither in PHD/nebulosity, but RA only (I have a portable system and set up every night, so PA is always a bit off - dither in dec upsets PHD, which works best with single side tracking to remove minor dec drift. If the dither throws the dec to the "no correction" side, it doesn't come back).

Best stacking for LRGB is average with default sigma rejection in Pixinsight - the normalisation routines in the PI stacking algorithms work really well with variable data. For NB, have found that dither plus Pixinsight median works best at cleaning up thermal fixed pattern noise - although it throws away some sensitivity, it still produces measurably better SNR than dark calibration to do the same.

[Paul Haese]

Sorry Mike you do need flats. In your recent image (http://www.pbase.com/strongmanmike2002/image/153065831/original)on the full frame top right of the image you can see either vignetting or the edge of a filter. That is a sure indication that you do need flats. Not to mention that dust motes are present on all optical surfaces. No imaging train is ever clean enough not to do flats. I find a discussion where flats are not used as being a form of laziness and not setting a good example to others who aspire to be a better imager. Not saying you are lazy but the idea of not doing flats when using good quality gear is totally abhorrent.

[Shiraz]

Even with small pixels, I have never found dust bunnies to be the slightest problem at f4 and with clean optics - probably quite a different story at f8. Dithering plus sigma rejection seems to get rid of any slightly darker small stuff as well as all the bright stuff, although I have had to use Nebulosity's hot pixel rejection on one set of subs when the dithering overlaid a few hot pixels and PI sigma reject could not quite handle it.

[gregbradley]

Thanks for the tips regarding Pix Insight. It seems to be the way to go. Martin mentioned PI gave the best stacking results compared to CCDstack although tests done on this site afterwards seemed different.

However I did try to use Lancosz 36 sampling after registration on a recent image in CCDstack and I found the images were misaligned and I have to do it all again with bicubic Bspline 36 where I have never had an issue.

I wouldn't mind a thread about flats at some point as well. My CDK17 is very sensitive to flats so I have found experimenting and taking advice from Don Goldman very helpful getting better flat fielding.

Greg.

[strongmanmike]

Thanks for the insight mate :rolleyes: Sooo, let's see?..I have averted imagination, do abhorrent things aaand I'm lazy...anything else :shrug: :lol:

All good Paul, I can live with it and all those aspiring imagers out there, make sure you do flats and take long subs and very long exposures... or Paul will explode :lol: (joking!)...actually I am a bit lazy :question:..must be getting old :P.

Mike

[gregbradley]

Perhaps a side by side comparison of an image with and without flats would be good.

Greg.

[Shiraz]

even better Greg, attached is an unprocessed flat from my system scaled to 8 bits. The variation across the frame is about 20% and it varies very smoothly, so for galaxy imaging, I just rely on the standard PI gradient removal process to take this out as well - which it does most effectively. As you can see, dust bunnies are not a problem at f4. Flat calibration may just possibly be useful if I ever decide to image something that fills the frame, but for now, there is simply no point in using it for galaxies and PNs.

In the absence of good weather, did some darks to test out the effect of dark calibration on a few combination methods. Using a small featureless central region of some dithered red subs of the Helix, I tried a variety of combination methods with the following results. Note SNR here is that that of the signal+sky in the chosen central region - 2 sigma outlier rejection was applied to the PI stacks and default normalisation was applied:

Median, no calibration - SNR=29.7 (PI)
0.1% Hot pixel reject - SNR=30.5 (Nebulosity)
Median with dark cal - SNR=31.3 (PI)
Average no calibration - SNR=33.6 (PI)
Average with dark cal - SNR=35.7 (PI)

The SNR difference between median and average is not quite the theoretical ~20%, but it is still reasonable for the experimental uncertainties. Simple hot pixel mapping works OK.

The choice is between the simplicity and high overall efficiency of the uncalibrated approaches vs the higher SNR of the calibrated approaches. Average stacking with no calibration and with outlier rejection seems to me to be the compromise that generally best suits my imaging and processing opportunities, although I have used calibration when trying to extract the last skerrick of info from a low brightness target (the difference was right on the edge of being perceptible).

I have not yet done a similar test on narrowband imaging.

[gregbradley]

Wow, great post Ray.

I did not know there was such a difference between average and median combining. I wonder if that is true for other cameras. I'll have to check that out.

I just received my Trius 694 camera and its clear out so hopefully I will be imaging with it tonight.

Greg.

[RickS]

The difference is due to the inherent nature of average vs median combination and is true for any camera.

[Shiraz]

further to Rick's comment, it is explained quite well here. http://pixinsight.com/doc/tools/ImageIntegration/ImageIntegration.html

Greg, hope it goes well with the 694. If you put it on the CDK, it will be waaay oversampled and consequently only 1/3 as sensitive as the 16803, but if you get sub arcsec seeing, the detail will be amazing.

[strongmanmike]

WELL I never!...that's just all so..abhorrant :eyepop: :lol: (just a light hearted joke Paul)

Mike