Flat noise reduction

[Amaranthus (Barry)]

Mike, why not just use Bad Pixel Mapping instead of Darks?

[Shiraz (Ray)]

thanks for the comments and discussion.

the demo image was formed when I wanted to isolate noise from signal structure in order to test the predictions of the model against what happens with real image data. I used 15 evenly illuminated lights (no structure) and then calibrated and stacked them as normal, with full flat/bias/dark calibration. When I wanted to investigate dithering, there was a need to move the noise pattern around in the lights so that, when they were stacked without alignment (nothing to align on), the noise would not be correlated from sub to sub - just like you get with dithering, only in this case the frame is fixed, but the noise was moved, rather than the noise being fixed, but the frame moving. It is not a practical method for real image processing, just something I had to use to simulate the effects of dither in featureless light frames.

Rolf, if you use dither, there is probably a lot of merit in removing the pixel scale noise in the flats with spatial filtering. I had been thinking along similar lines of a dual approach to flat fielding with one heavily smoothed flat from each imaging night to remove vignetting and a one-off separate master flat, prepared with even illumination and a lot of flats, to deal with fixed pattern noise and applicable to all images. Be very interested to hear how you get on with smooth flats to remove vignetting and dither to remove FPN. Also, if you are getting some additional noise from using full calibration, suggest that you can improve that with more flat, bias and dark data - the model suggests that in all cases, calibration is ultimately the best way to go. However, it also shows that you may need really high quality calibration data to get to the point where there is some extra SNR from calibration. In the end I guess that it comes down to which approach is more efficient and that will depend on the FPN in your camera. I also suspect that an outstanding problem is that darks contain a subset of warm pixels that generate significant current. You can remove the fixed component of that current by subtraction, but the noise associated with the higher current will remain much more significant than that from normal pixels - ie you may need a lot more darks than conventional wisdom would suggest to get rid of all the dark noise. One possible solution - haven't tried it yet- may be to use aggressive and identical hot pixel replacement to get rid of the worst dark noise sources in lights, darks and flats before doing any other processing. EDIT: as Barry suggests.

Mike - yep, your technique is very powerful - and it is statistically likely that you are not a pariah.

the model is now at the stage where I can probably expand it to include dark and bias. Might be interesting to try, but will probably be quite a job due to the odd statistics of dark noise. Do you think it could be worth publishing the model? - it clearly is not of major importance, but the results are quite interesting and I haven't seen anything like it elsewhere. if so, any ideas where?

[strongmanmike (Michael)]

Originally Posted by Shiraz

Mike - yep, your technique is very powerful - and it is statistically likely that you are not a pariah.

Nice to finally hear I have been told I am a heretic spreading bad habits

I think you should keep going with yoru analysis, some excellent useful info is being quantified...stuff that lazy bums like me avoid doing

Mike

[strongmanmike (Michael)]

Originally Posted by Amaranthus

Mike, why not just use Bad Pixel Mapping instead of Darks?

Do you mean for the H694 or the 16803?

Mike

[Amaranthus (Barry)]

Either - there is arguably little to be gained over a BPM, and much to be introduced (noise wise) by using Darks. The advantage of a BPM is that it will only fix (interpolate from surroundings) the bad pixels you designate (which you have control over, anywhere from 0 to 100s) and will leave every other pixel clean and untouched.

You really only have to match exposure and take a few frames to get a decent BPM, and even this is not critical like with Darks. In theory you could take a suite of long exposures (e.g. 10+ min), create a (bias-subtracted) Master Dark from these, and then just tune your BPM to match your actual exposure (fewer pixels for shorter subs). Keep the same Master Dark for months then, and forget about temperature matching etc.

[RickS (Rick)]

Ray,

Have you seen the new PI Superbias process? It uses multiscale analysis to isolate large scale structures and generate a "noise free" master bias from a lower quality master. Perhaps Superflat is next

Cheers,
Rick.

[Shiraz (Ray)]

Originally Posted by RickS

Ray,

Have you seen the new PI Superbias process? It uses multiscale analysis to isolate large scale structures and generate a "noise free" master bias from a lower quality master. Perhaps Superflat is next

Cheers,
Rick.

Yes, Marc pointed it out.

definitely seems to be related. I got the impression that the basis for the superbias approach is that bias will vary predictably (in a spatial sense) and that any abnormal pixel-to-pixel variation must be down to read noise - and hence removable. In the case of flats, the pixel-to-pixel variation can be FPN, flat read noise, shot noise or (darkshot+readdark+readbias) from the dark cal - this may be much less amenable to filtering. We live in interesting times though and it is good to see some formal optimisation of the processes we use.

[Amaranthus (Barry)]

Agree Ray, with caveats. If you use a light box and take really short-duration flats (I have a dimmer switch on mine and so can take nice 50% max ADU flats at a consistent 0.1 sec exposure), then bias subtract from those flats, it seems then well positioned to make a "super", no?

[RickS (Rick)]

Originally Posted by Amaranthus

Agree Ray, with caveats. If you use a light box and take really short-duration flats (I have a dimmer switch on mine and so can take nice 50% max ADU flats at a consistent 0.1 sec exposure), then bias subtract from those flats, it seems then well positioned to make a "super", no?

That won't work with a lot of mechanical shutters, unfortunately.

[Shiraz (Ray)]

Originally Posted by Amaranthus

Agree Ray, with caveats. If you use a light box and take really short-duration flats (I have a dimmer switch on mine and so can take nice 50% max ADU flats at a consistent 0.1 sec exposure), then bias subtract from those flats, it seems then well positioned to make a "super", no?

If very short subs are possible, that sounds like a good approach Barry - gets away from dark correction in making the flats, so lots of noise goes out the window too. Have you tried it? maybe also consider a median filter to remove any residual hot pixel data? Of course, would only work properly if you dither.

[Amaranthus (Barry)]

I don't have PI software, so can't try out their superbias process/algorithm (any idea how it works, exactly?). But I definitely only use bias subtraction in my ultra-short flats, since read noise will completely dominate over any (minimal) dark current noise that might arise in 0.1s.

Hot pixels effects should also be really minimal in 0.1s flat exposures, but one could remove them with a BPM and dithering if it was a concern.

[RickS (Rick)]

Originally Posted by Amaranthus

I don't have PI software, so can't try out their superbias process/algorithm (any idea how it works, exactly?).

A couple of the more interesting posts on Superbias are here:
http://pixinsight.com/forum/index.php?topic=7286.0
http://pixinsight.com/forum/index.php?topic=7312.0

[Shiraz (Ray)]

just tested an alternative method for flat fielding that seems to get around the introduced noise problem. PixInsight has a background estimation algorithm (DBE) that generates a smooth surface representing the sky background. I extracted such a background from a single starfield sub and then applied it as the master flat in the standard calibration process for a complete batch of subs. Worked fine to clear up some minor vignetting and the resultant calibrated data is flat to better than 1% across the frame. And of course there is no dark noise, no read noise, no shot noise and only a little residual bias subtraction noise in this type of flat.

Would not work on dust bunnies, but can possibly provide high quality control of vignetting without introducing much flat noise at all. Of course it will not remove fixed pattern noise, but dithering can take care of that.

Be interested in opinions on this technique - I have only tried it one dataset, but it worked OK on both the lum and RGB data..

[SpaceNoob (Chris)]

Originally Posted by Shiraz

just tested an alternative method for flat fielding that seems to get around the introduced noise problem. PixInsight has a background estimation algorithm (DBE) that generates a smooth surface representing the sky background. I extracted such a background from a single starfield sub and then applied it as the master flat in the standard calibration process for a complete batch of subs. Worked fine to clear up some minor vignetting and the resultant calibrated data is flat to better than 1% across the frame. And of course there is no dark noise, no read noise, no shot noise and only a little residual bias subtraction noise in this type of flat.

Would not work on dust bunnies, but can possibly provide high quality control of vignetting without introducing much flat noise at all. Of course it will not remove fixed pattern noise, but dithering can take care of that.

Be interested in opinions on this technique - I have only tried it one dataset, but it worked OK on both the lum and RGB data..

The DBE tool works quite well, ABE too. I did not bother with flats when using the FSQ106 @f5, using the CDK though I first started to notice the impact of dust bunnies.... DBE did reduce these slightly, but not enough to justify not using flats at longer focal lengths. I also found it useful for battling gradients from street lights and other additive sources via subtraction.

[Merlin66 (Ken)]

The DBE tool is very popular with solar imagers to help remove the hot spot sometimes found in Ha optical systems

[Shiraz (Ray)]

Originally Posted by SpaceNoob

The DBE tool works quite well, ABE too. I did not bother with flats when using the FSQ106 @f5, using the CDK though I first started to notice the impact of dust bunnies.... DBE did reduce these slightly, but not enough to justify not using flats at longer focal lengths. I also found it useful for battling gradients from street lights and other additive sources via subtraction.

Originally Posted by Merlin66

The DBE tool is very popular with solar imagers to help remove the hot spot sometimes found in Ha optical systems

Thanks Chris and Ken - appreciate the feedback.

I had used DBE for both subtraction and division on stacked data, but thought that it could generate synthetic flats from single frames, for use in pre-stack calibration, in the following circumstances:
- Where I wish to combine historical data sets in which the system configuration and target orientation varies between datasets and where flat data may be either minimal or non-existent - flat fielding can reduce the impact of abrupt background changes that cannot be handled post-stack and a synthetic flat may be the only way to do this,
- Where I have nebula images and the DBE process cannot correct the images after stacking - a single sky flat or even a dark sky image of a starfield may be used, along with dithering of the lights, to get close to the results expected from perfect flat fielding without the need for extensive flat data.

The primary disadvantage would be that the synthetic flat can only be valid if the background illumination is consistent over the field of view and of course it will not correct for dust bunnies. It will also not correct for fixed pattern noise. However, I thought that it may be a useful additional flat technique in circumstances where there is insufficient flat data to support a conventional approach. I tried it out by calibrating a recent dataset using only bias and a synthetic flat and it worked very effectively, reducing the gradient in vignetted lights to below 1% with only a single starfield image as the flat data source. dithering took care of dark and fixed pattern noise.

[RobF (Rob)]

I'm late to the party, but thought worth saying thanks Ray (and all) for a great thread. Also bumping this for anyone else that missed first time around. Take home message (shoot as many flats as lights) rings true.

[Shiraz (Ray)]

Originally Posted by RobF

I'm late to the party, but thought worth saying thanks Ray (and all) for a great thread. Also bumping this for anyone else that missed first time around. Take home message (shoot as many flats as lights) rings true.

thanks Rob - nice thought.

And thanks also for emphasising the primary message - use as many flats as lights, if you can. That message got a bit diluted in later discussion which included dithering (reduces the need for flats), but as a failsafe, equal numbers is a reliable approach.

[Rod771 (Rod)]

Originally Posted by Shiraz

just tested an alternative method for flat fielding that seems to get around the introduced noise problem. PixInsight has a background estimation algorithm (DBE) that generates a smooth surface representing the sky background. I extracted such a background from a single starfield sub and then applied it as the master flat in the standard calibration process for a complete batch of subs. Worked fine to clear up some minor vignetting and the resultant calibrated data is flat to better than 1% across the frame. And of course there is no dark noise, no read noise, no shot noise and only a little residual bias subtraction noise in this type of flat.

Would not work on dust bunnies, but can possibly provide high quality control of vignetting without introducing much flat noise at all. Of course it will not remove fixed pattern noise, but dithering can take care of that.

Be interested in opinions on this technique - I have only tried it one dataset, but it worked OK on both the lum and RGB data..

Geez, I like the sound of this. Taking flats through the Hyperstar lens with a DSLR and LP filter just sucks.

Ray, was the star field sub used for the synthetic flat exposed for the same time as your lights? And am I right in saying, that you only used one DBE extraction image as master flat and this was effective on a large number of light subs?

[Shiraz (Ray)]

Originally Posted by Rod771

Geez, I like the sound of this. Taking flats through the Hyperstar lens with a DSLR and LP filter just sucks.

Ray, was the star field sub used for the synthetic flat exposed for the same time as your lights? And am I right in saying, that you only used one DBE extraction image as master flat and this was effective on a large number of light subs?

yes, it was one of the light subs. The DBE background was then applied just like a normal master flat would be - (you must set downsample under model-image to 1 though). It gets rid of vignetting and even helps with the worst effects of large donuts (if you use enough sampling points). It does not do fixed pattern noise reduction, but dither can take care of most of that - it's not as good as a real flat, but still useful if you have problems getting flat data. You don't get a lot of signal in the synthetic flat, but, being a fitted surface, it is essentially noise free. if you do try it, be interesting to hear if it works well enough - I guess there will be issues with a DSLR due to the Bayer filter, but you should be able to find a way round them.