#1  
Old 25-07-2015, 09:44 AM
codemonkey's Avatar
codemonkey (Lee)
Registered User

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 1,770
Darks - count vs duration

Does the duration of exposures have any impact on the number of dark frames that you should use to generate a master?

For example, right now I'm capturing a series of 40min dark frames, as this is the max exposure time I use (with OIII). Being 40min long, would this have the effect of averaging out the noise to an extent?

I'm not even sure if it'll have any positive impact with the low noise sensor I have, but I need to build a defect map and I figured I't take the time to do some darks while I was at it and I'm not sure what a good number is.
Reply With Quote
  #2  
Old 25-07-2015, 11:12 AM
Slawomir's Avatar
Slawomir (Suavi)
Amateur Photon Collector

Slawomir is offline
 
Join Date: Sep 2014
Location: Proserpine
Posts: 3,023
Hi Lee,

When I was recently reevaluating my workflow it has been pointed out to me that there is a tool in PI that will generate noise free SuperBias from a MasterBias, and it worked very well for me. Taking a similar approach to MasterDark- I carefully experimented with noise reduction for my MasterDark and although I have not actually measured differences in the final image, it seems that it has not hurt my lights. Some would argue that only subtracting Bias is useful and that Darks are not needed, but my camera produces very faint gradients in longer darks (possibly due to some temperature variations across the CCD or amp glow?). Although these gradients are not showing up in light frames (only about 10-15 ADUs variations across the image in a 15-minute exposure ), I still like to do everything I can to eliminate any extra signal introduced by the camera (and the optical train with flats).

When I get time I will measure differences are report here.

S.
Reply With Quote
  #3  
Old 25-07-2015, 03:38 PM
codemonkey's Avatar
codemonkey (Lee)
Registered User

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 1,770
Cheers mate :-)

I've got a fair amount of hot/warm pixels in my camera so I was mostly just building up a master dark that I could use to generate a good defect map. Probably didn't need to match exposure length for this purpose, but I figured since I had the camera down I might as well..

I ended up doing 11x40min darks. I ran a quick test against some data I've recently acquired:

1 set of broadband data, 20x420s
1 set of narrowband data 9x1800s

The broadband data showed a measurable and visually apparent increase in noise.

The narrowband data showed a measurable and visually apparent decrease in noise.

Based on this, in future I'll be using darks on those long narrowband subs but I'll just use my defect map on the shorter subs.
Reply With Quote
  #4  
Old 25-07-2015, 03:44 PM
codemonkey's Avatar
codemonkey (Lee)
Registered User

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 1,770
Hmm, interesting. I went and double checked the temps I took those sets at, since I just remembered I'd been screwing with that lately. Neither set was taken at the temp I did the darks at.... well done, I must say lol.

Broadband data was done at -25, narrowband at -15, and darks at -20. Seems I'll need some more data before I can make a decision.

Forgot to mention; thanks for the tip on the superbias... I'd never really noticed it. I've done some reading on it and I'll probably use that going forward. My current bias master is made up of 100 frames, but it sounds like using superbias on it might be even better.
Reply With Quote
  #5  
Old 25-07-2015, 09:29 PM
Slawomir's Avatar
Slawomir (Suavi)
Amateur Photon Collector

Slawomir is offline
 
Join Date: Sep 2014
Location: Proserpine
Posts: 3,023
Hi Lee,

SuperBias will generate a noise free Bias frame that will only contain patterns introduced by the camera. So by subtracting correctly made SuperBias you will remove some camera-introduced patterns from your Lights without introducing additional noise. Then, if you only used a defect map you should end up with cleaner images. Noise in your Lights though should remain the same. I think we should keep in mind that subtracting perfect noise free Bias and noise free Dark frames will not bring the noise down in your Lights, but will only remove camera introduced artefacts. Bringing down noise is done by correct stacking of a number of subs, and optionally by noise reduction techniques in processing of the image. That's the way I see it anyway, and will be glad to be corrected if I am wrong.

EDIT: just to clarify, MasterDark contains Bias so you would not be subtracting both from your lights. However, in theory, calibrated/scaled MasterDark and MasterBias are both needed to properly prepare Flats.

Last edited by Slawomir; 26-07-2015 at 05:47 AM.
Reply With Quote
  #6  
Old 26-07-2015, 08:04 AM
codemonkey's Avatar
codemonkey (Lee)
Registered User

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 1,770
Cheers mate :-)

I know it's unusual, but dark subtraction on the narrowband set definitely did reduce the noise in the resulting integration, both visually and measurably.

The stddev was ~1094 (16-bit scale) for the dark-subtracted integration, and ~1216 for the integration without darks and I understand this to be a good measure of noise.

A couple of things of note there is that it also had a measurable difference in the image as well. An identical stretch looked quite different when applied to a small region in the resulting integrations. The median also differed by over 20% (529 with dark vs 420 without). This had the effect of decreasing global contrast, which you can see in the attached crops which were stretched identically.

Dark subtraction is known to increase noise in the integration, thus the reason I was comparing integrations with dark, without dark and without dark but with defect map, but these results aren't what I would expect.

I wonder if this is related to the mismatch in temperature, with the darks being shot at 5c lower than the lights... I would have expected this to increase the noise if anything...
Attached Thumbnails
Click for full-size image (dark.jpg)
79.0 KB22 views
Click for full-size image (no_dark.jpg)
85.6 KB17 views
Reply With Quote
  #7  
Old 26-07-2015, 09:24 AM
Slawomir's Avatar
Slawomir (Suavi)
Amateur Photon Collector

Slawomir is offline
 
Join Date: Sep 2014
Location: Proserpine
Posts: 3,023
Hi Lee,

Interesting results indeed.

I would be careful though when comparing StDev for the entire images and using it as a measure of noise. Dark subtracted image will have lower mean ADU value, and that could affect the StDev? Perhaps instead of comparing StDev for the entire images, pick a smallish area with background only, if possible, and check StDev for this area in both images.

Anyway, whatever technique works and if you are ending up with cleaner images, that's great
Reply With Quote
  #8  
Old 26-07-2015, 02:28 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,787
since your NB lights are almost identical to your darks (with some added nebula and stars, but with very little background sky), getting rid of dark Fixed Pattern Noise is essential. Dark subtraction is the most effective way to do that in narrow band, where there is no sky noise to hide the read and thermal noise. Dark subtraction fixes FPN, but introduces random noise (read and shot), so, to make quite sure that you get an advantage (ie the noise reduction from fixing FPN outweighs the increase in random noise), I think that it is probably prudent to have about 3x as many darks as lights. You will get OK results with less, but there will still be a little to gain by using as many as you possibly can. If you must use limited dark data, be sure to dither - that will help.

The other thing that you might try is to do a dark subtraction and then follow by a hot pixel substitution - this will get rid of shot noise associated with warm and hot pixels (which can show up as minor dark and bright spots even after dark subtraction).

If you use PI to calibrate, it will make optimum use of whatever dark data you have - even if it does not exactly match the light exposures.

Last edited by Shiraz; 26-07-2015 at 02:44 PM.
Reply With Quote
  #9  
Old 26-07-2015, 07:16 PM
codemonkey's Avatar
codemonkey (Lee)
Registered User

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 1,770
Quote:
Originally Posted by Slawomir View Post
Hi Lee,

Interesting results indeed.

I would be careful though when comparing StDev for the entire images and using it as a measure of noise. Dark subtracted image will have lower mean ADU value, and that could affect the StDev? Perhaps instead of comparing StDev for the entire images, pick a smallish area with background only, if possible, and check StDev for this area in both images.

Anyway, whatever technique works and if you are ending up with cleaner images, that's great
Hmm, fair enough. Hard to find background in the data set that I have unfortunately since it's full of faint nebulosity. Cheers mate!

Quote:
Originally Posted by Shiraz View Post
since your NB lights are almost identical to your darks (with some added nebula and stars, but with very little background sky), getting rid of dark Fixed Pattern Noise is essential. Dark subtraction is the most effective way to do that in narrow band, where there is no sky noise to hide the read and thermal noise. Dark subtraction fixes FPN, but introduces random noise (read and shot), so, to make quite sure that you get an advantage (ie the noise reduction from fixing FPN outweighs the increase in random noise), I think that it is probably prudent to have about 3x as many darks as lights. You will get OK results with less, but there will still be a little to gain by using as many as you possibly can. If you must use limited dark data, be sure to dither - that will help.

The other thing that you might try is to do a dark subtraction and then follow by a hot pixel substitution - this will get rid of shot noise associated with warm and hot pixels (which can show up as minor dark and bright spots even after dark subtraction).

If you use PI to calibrate, it will make optimum use of whatever dark data you have - even if it does not exactly match the light exposures.
Thanks Ray, lots of good info there, as always :-)
Reply With Quote
  #10  
Old 27-07-2015, 07:52 AM
Terry B's Avatar
Terry B
Country living & viewing

Terry B is offline
 
Join Date: Mar 2006
Location: Armidale
Posts: 2,761
Very long exp darks will need more sub exposures compared to short exposures for a different reason. Assuming that cosmic ray hits appear at a reasonably constant but random rate, the longer the total exposure is, the more likelihood that they will affect your dark exposures. They are removed essentially when you combine your darks but are not part of the dark current and can leave residual noise. The more exposures you have, the better they are removed.
Shorter exposures will have less hits and as 10x 30 sec frames is less total exposure than 10x 20min exposures you can get away with less subframes for the same noise in the final master dark.
Reply With Quote
  #11  
Old 27-07-2015, 12:27 PM
Slawomir's Avatar
Slawomir (Suavi)
Amateur Photon Collector

Slawomir is offline
 
Join Date: Sep 2014
Location: Proserpine
Posts: 3,023
Hi Lee,

For comparing noise in deferent images, selected area does not necessarily need to be just the background. Maybe pick a region that is fairly uniform and with low signal.

I still believe in the future of noise reduction being skilfully applied to Master Darks
Reply With Quote
  #12  
Old 27-07-2015, 04:13 PM
codemonkey's Avatar
codemonkey (Lee)
Registered User

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 1,770
Quote:
Originally Posted by Terry B View Post
Very long exp darks will need more sub exposures compared to short exposures for a different reason. Assuming that cosmic ray hits appear at a reasonably constant but random rate, the longer the total exposure is, the more likelihood that they will affect your dark exposures. They are removed essentially when you combine your darks but are not part of the dark current and can leave residual noise. The more exposures you have, the better they are removed.
Shorter exposures will have less hits and as 10x 30 sec frames is less total exposure than 10x 20min exposures you can get away with less subframes for the same noise in the final master dark.
Interesting post, thanks Terry! You've actually inadvertently answered a question I had regarding these seemingly random squiggles that appeared in these long darks. After googling about cosmic rays and CCDs I now have the answer.

I think I'll be capturing some more darks in the near future for these long narrowband subs if nothing else.

Quote:
Originally Posted by Slawomir View Post
Hi Lee,

For comparing noise in deferent images, selected area does not necessarily need to be just the background. Maybe pick a region that is fairly uniform and with low signal.

I still believe in the future of noise reduction being skilfully applied to Master Darks
Ah ok, fair enough. Interesting idea using NR in master darks... I've never actually thought of that.
Reply With Quote
  #13  
Old 27-07-2015, 05:38 PM
Slawomir's Avatar
Slawomir (Suavi)
Amateur Photon Collector

Slawomir is offline
 
Join Date: Sep 2014
Location: Proserpine
Posts: 3,023
Quote:
Originally Posted by codemonkey View Post
Ah ok, fair enough. Interesting idea using NR in master darks... I've never actually thought of that.
I think it is worth trying. In the end, with Dark and Bias frames, we aim for separating pure signal introduced by the camera from noise, in a quite similar fashion as with Lights, where we would like to have sky data without noise added by numerous factors.
Reply With Quote
  #14  
Old 27-07-2015, 07:35 PM
Slawomir's Avatar
Slawomir (Suavi)
Amateur Photon Collector

Slawomir is offline
 
Join Date: Sep 2014
Location: Proserpine
Posts: 3,023
Apologies for a longish post...

Did a quick test to see if there were any measurable differences in using "regular" Master Dark (made from 28 subs of 900 s duration) and Master Bias (created from 200 subs) frames, as opposed to a noise-free Bias (created with PI SuperBias tool) and a near noise-free MasterDark (manually reduced noise).

I selected three different sections of a Ha image (900 s exposure), one containing low levels of signal (background), one containing some signal and one with high levels of signal and measured StDev for these three areas.

Then, after calibrating the same Ha image with MasterBias and calibrated MasterDark, I measured StDev for the same three areas in the new image. The process was repeated for the same sub "treated" with noise-free Bias and near noise-free Dark.

The differences, as expected, were miniscule...

Applying standard master calibration frames increased StDev in the measured area by 0.004% (background), 0.001% (some signal) and 0.03% (high signal) while applying SuperBias and calibrated noise-reduced Dark resulted in a decrease in StDev by 0.130% (background) and by 0.058% (some signal) and in an increase in StDev by 0.002% in high signal area.

Decrease in StDev for the two areas in the Ha image calibrated with noise-free Bias and calibrated near noise-free Dark could be explained by having a smaller mean ADU values. Since applying "regular" MasterBias and calibrated MasterDark also decreased mean ADU while sightly increasing StDev, I would suggest that the overall difference between the two techniques results in about 0.07% (rounding off averages for all three areas in both images) in favour of using noise free calibration frames.

Obviously further testing is required to provide with higher confidence estimates.
Reply With Quote
  #15  
Old 28-07-2015, 02:40 PM
codemonkey's Avatar
codemonkey (Lee)
Registered User

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 1,770
No need to apologise, I appreciate the longish post! :-)

I wonder why we both got such different results... yours does seem to be closer to what one would expect given conventional knowledge.

I'm inferring from your post that you applied the calibration frames to one image? I applied mine to a set and then integrated them before my analysis... that could at least partially explain it.

I've had the camera running for a few hours today getting more 40min darks. I'll do some more tests in the coming days and post my results.
Reply With Quote
  #16  
Old 28-07-2015, 05:24 PM
Slawomir's Avatar
Slawomir (Suavi)
Amateur Photon Collector

Slawomir is offline
 
Join Date: Sep 2014
Location: Proserpine
Posts: 3,023
Hi Lee,

I thought I killed your thread with my last post...

Yes, I used only a single sub for testing. Your method of applying calibration frames to a set of subs is superior to mine as you basically have a larger number of samples for more accurate statistical analysis. But again, I think that measuring StDev for parts of the image, in particular in the areas of low signal, gives a better indication of noise as opposed to comparing StDev for the entire images.

More darks and more bias is always better, but I feel that after certain point you can successfully apply noise reduction techniques to MasterBias and MasterDarks and get a very similar result as with a significantly larger number of calibration frames taken to create less noisy Master Bias and Dark frames.

It essence, Master calibration frames can/should be approached in the same way as one would work with the "normal subs" (Lights) - you want a sufficient number of subs to have a good signal and then you apply noise reduction without distorting your "real" data (signal), and that data (signal we are after) can be from the sky (Lights) or from the camera (Bias and Darks).

Oh no, I have been waffling again...
Reply With Quote
  #17  
Old 28-07-2015, 06:25 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 15,480
Wouldn't noise reduction on the darks lower the values for the hot pixels making them not subtract completely?

Greg.
Reply With Quote
  #18  
Old 28-07-2015, 06:53 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,787
Quote:
Originally Posted by gregbradley View Post
Wouldn't noise reduction on the darks lower the values for the hot pixels making them not subtract completely?

Greg.
I think that is the case - any form of spatial noise reduction on the darks will actually make the situation worse - the only thing that will effectively reduce read and shot noise in the master dark is lots of dark subs. Same applies to flats if you are relying on them for FPN reduction.
Reply With Quote
  #19  
Old 28-07-2015, 07:48 PM
Slawomir's Avatar
Slawomir (Suavi)
Amateur Photon Collector

Slawomir is offline
 
Join Date: Sep 2014
Location: Proserpine
Posts: 3,023
Thank you Ray and Greg - I did not think that carefully applied noise reduction would lower values for the hot pixels - will need to investigate that more closely. Anyway, I like exploring off the beaten track, even if only leads me back to the old and well established ways.
Reply With Quote
  #20  
Old 28-07-2015, 07:59 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 15,480
Quote:
Originally Posted by Slawomir View Post
Thank you Ray and Greg - I did not think that carefully applied noise reduction would lower values for the hot pixels - will need to investigate that more closely. Anyway, I like exploring off the beaten track, even if only leads me back to the old and well established ways.
I think so Slawomir as noise reduction usually is merely a blur which would be something like a running average which would lower the values of the hot pixels and raise the values of the nearby surrounding pixels.

The perfect dark would be an exact duplicate of the light minus the signal.

Greg.
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 12:40 AM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Advertisement
EQ8-R
Advertisement
Astromechanics
Advertisement
OzScopes Authorised Dealer
Advertisement
Lunatico Astronomical
Advertisement
Bintel
Advertisement
Star Adventurer
Advertisement
Celestron RASA
Advertisement
Astronomy and Electronics Centre
Advertisement