#1  
Old 14-06-2014, 04:13 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Flat noise reduction

Hi

Just finished developing a model of the way in which flats introduce noise into broadband images. Also did a validation using some real data, to be reasonably sure that the model describes reality. This analysis has nothing at all to do with removal of vignetting or dust donuts - just noise reduction. If flats are good enough to fix noise they will take care of other stuff.

Real world results in the first figure show the way in which image noise varies with total flat signal - these four images were obtained using the same sequence of real images (totaling just under 4 hours with dark sky), but with different flats exposure. The data has been processed using flat data where the total flat signal was 0.05, 0.5, 2.7 and 11x the total sky signal. The images show the effects of noise reduction as the total flat exposure goes up. Note that the noise shows up as lines since the imaging system had some drift - this makes any fixed pattern noise stand out well and shows how this form of noise affects bright parts of the image as well as the background (it really is a nasty form of noise). Clearly there is some advantage in having a reasonably long total flat exposure - the fourth image has flat exposure as recommended by the model and would need more light exposure to reduce noise further (and of course longer flat exposure).

The real world SNR results from the 4 images are compared with model results in the second figure. Model data is shown as a dashed line and the four measured results as diamonds. The model results are "worst case", so the agreement between model and real world seems convincing enough (at least to me) for the model to be used to derive some general guidance on how to expose flats. It shows that, if you are doing broadband imaging, flat induced noise will be negligible if you get at least 10x as many flat electrons as you get total sky electrons. This very simple rule will apply under all circumstances and can be used to replace ad-hoc guessing, rumor or one-off observations. Since the model is "worst case", you will probably be able to get by with fewer flats most of the time, but flats are easy to obtain, so it makes sense to use enough to ensure that you will not be troubled by flat noise in any circumstances.

If you want an even easier rule of thumb that will get you close enough, expose your light subs properly (eg about 2-3000 ADU in the sky background) and then make sure that you take at least one flat of 20-30,000 ADU for each light sub. If you take sky flats with widely varying exposures, adjust the number of flats to maintain the total ADU over the set.

The graph also shows the measured SNR for no flats (a single dot on the y axis) - with ~4 hours of lights and a very low noise CCD (in this case an icx-694), there is not much point in using flats at all. This is definitely not the case for much longer total exposures, or for CCDs with a little more fixed pattern variabilty (eg some of the Kodak ones) - then flats are necessary.

Thanks for looking. Regards ray

Added: for anyone looking for the last skerrick of SNR, the 10x rule gets you within 5% of the SNR you would get with infinite flats - but you can always squeeze a little more SNR by using more flat data.
Attached Thumbnails
Click for full-size image (noise.jpg)
139.7 KB184 views
Click for full-size image (noisegraph.jpg)
26.2 KB147 views

Last edited by Shiraz; 16-06-2014 at 01:08 PM.
Reply With Quote
  #2  
Old 14-06-2014, 07:19 PM
Octane's Avatar
Octane (Humayun)
IIS Member #671

Octane is offline
 
Join Date: Dec 2005
Location: Canberra
Posts: 11,159
Excellent work, Ray. This is greatly appreciated.

H
Reply With Quote
  #3  
Old 14-06-2014, 08:59 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by Octane View Post
Excellent work, Ray. This is greatly appreciated.

H
Thanks H. It's fun to try to find a theoretical basis for some of the stuff we do - hope it is of some use. regards Ray
Reply With Quote
  #4  
Old 14-06-2014, 09:05 PM
Octane's Avatar
Octane (Humayun)
IIS Member #671

Octane is offline
 
Join Date: Dec 2005
Location: Canberra
Posts: 11,159
I got my RoboFocus back from the US. It's all fixed. But, do you think there's been a break in the clouds and rain since it arrived?

I'm holed up inside going stir crazy; playing around with Skytools!

H
Reply With Quote
  #5  
Old 14-06-2014, 09:09 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 17,871
Thanks for that Ray, very helpful. The numbers on those images - what do they represent?

Greg.
Reply With Quote
  #6  
Old 14-06-2014, 09:12 PM
Octane's Avatar
Octane (Humayun)
IIS Member #671

Octane is offline
 
Join Date: Dec 2005
Location: Canberra
Posts: 11,159
The ADU of the flat vs the ADU in the signal of the light.

Hence, the 20,000-30,000 ADU value for the flat vs the 2,000-3,000 ADU for the light.

About 10x difference.

The 0.05 would represent 1,000-1,500 ADU flat, and, so on.

H
Reply With Quote
  #7  
Old 14-06-2014, 09:37 PM
trent_julie's Avatar
trent_julie
Registered User

trent_julie is offline
 
Join Date: Nov 2008
Location: Canberra
Posts: 581
Ray,
I got something out of this thanks for posting!

Trent
Reply With Quote
  #8  
Old 14-06-2014, 10:27 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by Octane View Post
I got my RoboFocus back from the US. It's all fixed. But, do you think there's been a break in the clouds and rain since it arrived?

I'm holed up inside going stir crazy; playing around with Skytools!

H
Quote:
Originally Posted by gregbradley View Post
Thanks for that Ray, very helpful. The numbers on those images - what do they represent?

Greg.
Quote:
Originally Posted by Octane View Post
The ADU of the flat vs the ADU in the signal of the light.

Hence, the 20,000-30,000 ADU value for the flat vs the 2,000-3,000 ADU for the light.

About 10x difference.

The 0.05 would represent 1,000-1,500 ADU flat, and, so on.

H
Quote:
Originally Posted by trent_julie View Post
Ray,
I got something out of this thanks for posting!

Trent
Thanks very much for the feedback guys. - H, thanks for the explanation ....clouds, arrgh.

Greg, the model shows that, for the flat noise to be under control, the sum of the ADU in the flats has to be >10x the sum of the sky ADU in the lights.

The four images show what I got from a single test set of lights (about 4 hours of subs), after I calibrated and stacked them four times using four different sets of flats. The sets of flats had total ADUs of 0.05x, 0.5x, 2.7x and 11x the total of the sky ADU in the lights - the number on an image shows which set of flats was used. These real life results suggest that the model is not complete crap, since the fixed pattern noise is visible in the images where the flat/sky ratio is 0.05x, 0.5x and 2.7x, but has effectively gone in the image where the flats sum is 11x the sky sum - just as the model says it should. The graph just summarises these results in numerical SNR form, rather than imagery.

To put some numbers on it, if you choose to take 50 light subs and the sky in each is 2,000 ADU, you will have a total of 100,000 sky ADU. You will need 10x that much (or 1,000,000) in the flats, so if your flats each have an ADU of 20,000, you will need at least 50 of them to get to the required 1,000,000 ADU total.

There is no fixed number for how many flats you should take - that will depend on how long you image for, how bright the sky is and how you expose the flats. But, provided you keep the total flat ADU to at least 10x the total sky ADU in the lights, the flat noise will always be negligible. Or, as a straightforward rule of thumb that will work well enough (for LRGB imaging), if you keep your lights around 2-3,000 ADU and your flats around 20-30,000 ADU, then you should:

******

Take at least one well exposed flat for every well exposed light sub that you take.

******



I know that I tend to add confusion when trying to explain things - be grateful if anyone would point out inconsistencies or lack of clarity in what has been said. Although the model itself is slightly arcane, the guidance that it provides is remarkably straightforward - I hope that is coming across.

regards ray

Last edited by Shiraz; 16-06-2014 at 01:03 PM.
Reply With Quote
  #9  
Old 15-06-2014, 07:30 AM
PRejto's Avatar
PRejto (Peter)
Registered User

PRejto is offline
 
Join Date: Jan 2011
Location: Rylstone, NSW, Australia
Posts: 1,397
Hi Ray,

Thanks so much for this work. I think it will really benefit my images if I follow this rule of 10X!!

Question. Some time ago someone on this forum suggested that I take flats at higher adu than the typical 20-30k adu. The claim was that fainter stuff would be visible. So, say if I aimed for 40,000 adu flats what does this do? And, in your opinion is there an advantage or disadvantage to such an approach?

Thanks!
Peter
Reply With Quote
  #10  
Old 15-06-2014, 08:15 AM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by PRejto View Post
Hi Ray,

Thanks so much for this work. I think it will really benefit my images if I follow this rule of 10X!!

Question. Some time ago someone on this forum suggested that I take flats at higher adu than the typical 20-30k adu. The claim was that fainter stuff would be visible. So, say if I aimed for 40,000 adu flats what does this do? And, in your opinion is there an advantage or disadvantage to such an approach?

Thanks!
Peter
Thanks Peter - yes, I am going to have to go back over some past data where I mucked up the flat calibration .

As far as I can tell, the only two conditions for flats is that they have to be taken in the linear region of the CCD and it makes sense to get well above the read noise. Depends on the CCD, but those conditions should pretty much always be met if the ADU is between 10,000 and 40,000. In terms of the final result, 40 flats of 10,000 ADU should be identical to 10 flats of 40,000 ADU. The only ways I can think of that this could break down is if you use stacking software that only has 16 bit fixed point internal data representation (I don't think anything modern is in that category), or you use median stacking for the flats (but why would you?). regards ray

Last edited by Shiraz; 15-06-2014 at 08:43 AM.
Reply With Quote
  #11  
Old 15-06-2014, 08:24 AM
PRejto's Avatar
PRejto (Peter)
Registered User

PRejto is offline
 
Join Date: Jan 2011
Location: Rylstone, NSW, Australia
Posts: 1,397
Thanks Ray!

So, if I understand correctly (assuming 40K adu are still "linear") using 40,000 adu flats becomes a rule of ca 5X (or ca 1 flat/2 images)

Peter
Reply With Quote
  #12  
Old 15-06-2014, 08:47 AM
Merlin66's Avatar
Merlin66 (Ken)
Registered User

Merlin66 is offline
 
Join Date: Oct 2005
Location: Junortoun Vic
Posts: 8,904
Ray,
Why not median combine?
It's only a scaling factor and doesn't change the SNR?
How do you measure your SNR?
I use average combine for my spectroscope lights and median for the darks...I haven't found flats very useful with the ATiK314 in this application.
Reply With Quote
  #13  
Old 15-06-2014, 08:50 AM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by PRejto View Post
Thanks Ray!

So, if I understand correctly (assuming 40K adu are still "linear") using 40,000 adu flats becomes a rule of ca 5X (or ca 1 flat/2 images)

Peter
10x still applies. You just need half as many 40,000 ADU flats to get to 10x, so take one 40,000 ADU flat for every 2 subs (of 2,000 ADU).
Reply With Quote
  #14  
Old 15-06-2014, 09:18 AM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 17,871
Thanks H and Ray for explaining what the numbers on the images meant.

Does this model assume a particular method of combining? Usual practice is a simple mean or average combine. I have stared using sigma reject combine but if I understand the statistical maths involved that method really requires a substantial number of flats to be effective as it tries to identify outliers that are outside the statistical average or norm. You can't do that with 3 subs.

I sometimes find flats that are too bright are harsh and damage the lights.

Also I assume there is no bias or flat dark subtract here? I have found no flat dark and using a separate bias gets the best results.

Greg.
Reply With Quote
  #15  
Old 15-06-2014, 09:30 AM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 17,871
I just checked a few subs I have. With my Microline 8300 the background seemed to be around 3300 (I simply moved the cursor around the image in CCDstack and the ADU is displayed in the bottom right corner).

For my Proline 16803 it was more like 2100.

So if I took 20 luminance subs like that with the Microline that is 20 x 3300=66,000. 10X rule gives me 10 x 66,000 = 660,000. If my flat subs are around 30,000 ADU then I need 660,000 divided by 30,000 =22 flats.
So an approximate rule there would be 1 flat for every light. This is all 1x1 binning.

For the Proline though it would need 1/3rd as many so say 15 flats for every 20 lights.

I did approximately 12 flats in my last imaging run.
But if I increase my subexposure length to 30 minutes I'll have to measure the background ADU (not sure if it increases with exposure length). It may mean longer exposures require less number of flats?
Another argument for longer exposures so long as bright areas don't blow out?

Greg.
Reply With Quote
  #16  
Old 15-06-2014, 09:31 AM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by Merlin66 View Post
Ray,
Why not median combine?
It's only a scaling factor and doesn't change the SNR?
How do you measure your SNR?
I use average combine for my spectroscope lights and median for the darks...I haven't found flats very useful with the ATiK314 in this application.
Hi Ken.

As I understand it, median combine will pull out the single value that is the median of all of the inputs - it cannot have a fractional ADU value, so the scaling factor will be subject to quantisation noise.

For SNR, the basic question is "of what". To fix a consistent reference for the modelling and measurements, I measured the average signal of a dim part of the galaxy to be 1/10 the sky background on one of the lights. Since the sky and the galaxy did not change much during the imaging run, I thereafter used 1/10 sky as the Signal and so the SNR values refer to the chosen part of the galaxy. I could choose some other reference point and all of the SNR results would be scaled, but the same picture would emerge when comparing model and test data. With a dim target, the noise is totally dominated by the sky, so I measured the RMS variability of a featureless part of the sky to find the noise. I turned off all of the smart bits while stacking in PI (to get an undoctored result) and used the pixel stats function of Nebulosity to do the measurements.

Interesting observation on the 314. I would guess that the Sony chip in the Atik has very low fixed pattern variability, so flats will not help unless you expose for very long periods of time.

Regards Ray

Last edited by Shiraz; 15-06-2014 at 11:05 AM.
Reply With Quote
  #17  
Old 15-06-2014, 09:43 AM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by gregbradley View Post
Thanks H and Ray for explaining what the numbers on the images meant.

Does this model assume a particular method of combining? Usual practice is a simple mean or average combine. I have stared using sigma reject combine but if I understand the statistical maths involved that method really requires a substantial number of flats to be effective as it tries to identify outliers that are outside the statistical average or norm. You can't do that with 3 subs.

I sometimes find flats that are too bright are harsh and damage the lights.

Also I assume there is no bias or flat dark subtract here? I have found no flat dark and using a separate bias gets the best results.

Greg.
Quote:
Originally Posted by gregbradley View Post
I just checked a few subs I have. With my Microline 8300 the background seemed to be around 3300 (I simply moved the cursor around the image in CCDstack and the ADU is displayed in the bottom right corner).

For my Proline 16803 it was more like 2100.

So if I took 20 luminance subs like that with the Microline that is 20 x 3300=66,000. 10X rule gives me 10 x 66,000 = 660,000. If my flat subs are around 30,000 ADU then I need 660,000 divided by 30,000 =22 flats.
So an approximate rule there would be 1 flat for every light. This is all 1x1 binning.

For the Proline though it would need 1/3rd as many so say 15 flats for every 20 lights.

I did approximately 12 flats in my last imaging run.
But if I increase my subexposure length to 30 minutes I'll have to measure the background ADU (not sure if it increases with exposure length). It may mean longer exposures require less number of flats?
Another argument for longer exposures so long as bright areas don't blow out?

Greg.
Hi Greg.

The assumption is average combine with floating point representation. It is assumed that outliers are taken care of using some non-linear technique that does not interfere significantly with the averaging process - sigma rejection would fit the bill.

This analysis does not include darks - it is assumed that dark calibration is perfect.

So an approximate rule there would be 1 flat for every light. This is all 1x1 binning. For the Proline though it would need 1/3rd as many so say 15 flats for every 20 lights.
why not stick with "1 flat for every light" - that will be close enough.

If you choose to deliberately overexpose the subs by increasing the exposure length, the sky background will increase in proportion to the time - you will still need the same number of flats to get to 10x the total signal, so there is no argument in favour of longer subs. However, the simple rule of thumb that you need one flat for every light will not apply if you deliberately overexpose - you will then need more than one flat for every light. The basic idea makes intuitive sense - if you expose for longer on the lights you need to expose for longer on the flats. It doesn't matter how you go about doing it - just how long the total sums of the exposures are.

I sometimes find flats that are too bright are harsh and damage the lights I don't understand what this means Greg - what sort of damage do you see?

regards ray

Last edited by Shiraz; 16-06-2014 at 01:15 PM.
Reply With Quote
  #18  
Old 15-06-2014, 11:49 AM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 17,871
I sometimes find flats that are too bright are harsh and damage the lights I don't understand what this means Greg - what sort of damage do you see?

regards ray[/QUOTE]


I'd have to look over past callibrations. Mostly with the CDK which has been the touchy one to flat out in the past. The APOs are a piece of cake and very forgiving. I don't have exact and precise steps I took at the time so forget the comment as I may not be able to replicate it but I would still say though that too bright a flat could be damaging. After all you are dividing the values into the light to even them out so too high a number from the flat could divide too savagely?? I could be wrong here.

Greg.
Reply With Quote
  #19  
Old 15-06-2014, 01:13 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by gregbradley View Post
I sometimes find flats that are too bright are harsh and damage the lights I don't understand what this means Greg - what sort of damage do you see?

regards ray


I'd have to look over past callibrations. Mostly with the CDK which has been the touchy one to flat out in the past. The APOs are a piece of cake and very forgiving. I don't have exact and precise steps I took at the time so forget the comment as I may not be able to replicate it but I would still say though that too bright a flat could be damaging. After all you are dividing the values into the light to even them out so too high a number from the flat could divide too savagely?? I could be wrong here.

Greg.
OK. I guess you might run into problems if you have to correct significant vignetting and the flats are near the non-linear region - guess that would depend on the software.
Reply With Quote
  #20  
Old 15-06-2014, 01:52 PM
Screwdriverone's Avatar
Screwdriverone (Chris)
I have detailed files....

Screwdriverone is offline
 
Join Date: Feb 2008
Location: Kellyville Ridge, NSW Australia
Posts: 3,306
Hi Ray,

Greg mentioned he uses the cursor in CCDStack to find the ADU value, although, as I use an Atik as well and am a bit of a noob still when it comes to taking flats with my light box, what do you suggest I use to determine the ADU value of the fits files? I think it works in Nebulosity when I debayer the fits file and go to the info page, but I am not confident this is correct?

Any tips on determining the correct ADU per flat would be of great help, as I find some flats (say 0.5 sec with my light box) give me a histogram in Nebulosity between say 18,000 and 30,000 so I guess that is somewhere in the 24,000 range for ADU?

Is there an easier way?

Thanks

Chris
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 05:53 PM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Advertisement
Bintel
Advertisement
Testar
Advertisement