#1  
Old 10-10-2013, 09:03 AM
rogerg's Avatar
rogerg (Roger)
Registered User

rogerg is offline
 
Join Date: Feb 2006
Location: Perth, Western Australia
Posts: 4,542
Mixing Lum exposures of diff lengths

Hi all,

Given the following:

30 x 180s L
20 x 300s L

exposures of a deep sky object, where neither image saturate the chip (so HDR not required) can anyone recommend the best way to utilise the data? I wouldn't want to just average all 50 frames because that would bring down the brightness of the 300s exposures, but I also don't want to waste the 180s exposures which could even out the noise in background areas.

Advice?

Regards,
Roger.
Reply With Quote
  #2  
Old 10-10-2013, 09:29 AM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,584
Integrate them with PixInsight using normalization Additive with scaling and weighted by noise evaluation
Reply With Quote
  #3  
Old 10-10-2013, 10:14 AM
rogerg's Avatar
rogerg (Roger)
Registered User

rogerg is offline
 
Join Date: Feb 2006
Location: Perth, Western Australia
Posts: 4,542
Quote:
Originally Posted by RickS View Post
Integrate them with PixInsight using normalization Additive with scaling and weighted by noise evaluation
Hmm, interesting.

Currently I don't have access to PixInsight. My trial expired and being 32bit OS it's hard to justify buying an unsuported version of it.

I'm currently trialling CCDStack. It can normalise exposures, and I would use sum/additive stacking. That just leaves the "scaling and weighted by noise evaulation" for me to work out an alternative
Reply With Quote
  #4  
Old 10-10-2013, 10:33 AM
alocky's Avatar
alocky (Andrew lockwood)
PI popular people's front

alocky is offline
 
Join Date: Aug 2010
Location: perth australia
Posts: 1,287
Hi roger - if you want to Dropbox the subs to me I'll cheerfully push it through pixinsight for you. Not a long term solution I know, but a quick fix.
Cheers,
Andrew.
Reply With Quote
  #5  
Old 10-10-2013, 10:41 AM
rogerg's Avatar
rogerg (Roger)
Registered User

rogerg is offline
 
Join Date: Feb 2006
Location: Perth, Western Australia
Posts: 4,542
Quote:
Originally Posted by alocky View Post
Hi roger - if you want to Dropbox the subs to me I'll cheerfully push it through pixinsight for you. Not a long term solution I know, but a quick fix.
Cheers,
Andrew.
Thanks for your offer Andrew. I have various objects which I have data like this for and some ongoing trying to get more .... so probably need to just work out a way to do it myself ... but might take you up on your offer once I have a complete set.
Reply With Quote
  #6  
Old 10-10-2013, 10:44 AM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,584
I really like the ability to use a noise estimate for weighting, Roger. It makes mixing mismatched data work quite well. I have even integrated a bunch of narrowband and RGB master frames to make a "superluminance" for a narrowband/RGB hybrid image.

You probably have too much data for it to be convenient, but if you want me to try a noise weighted integration so you can see the result just drop me a PM. [Whoops, just saw Andrew's offer after I hit the button to post my reply...]

Good luck!

Cheers,
Rick.
Reply With Quote
  #7  
Old 10-10-2013, 11:42 AM
algwat (Alan)
Registered User

algwat is offline
 
Join Date: May 2012
Location: Melbourne, west
Posts: 284
Gidday ,

Try imageJ, create a stack of all the images.
use the drop menu selection image>>stacks>>Zproject.
Select either sum, average, medium, or maximum.
Adjust the contrast and brightness
drop menu selection image>>adjust>>bright-contrast.

Should be ok for Lum iimages, a bit tricky for RGB sets.

fin imageJ here, http://imagej.nih.gov/ij/index.html

kind regards, Alan
Reply With Quote
  #8  
Old 10-10-2013, 12:50 PM
alocky's Avatar
alocky (Andrew lockwood)
PI popular people's front

alocky is offline
 
Join Date: Aug 2010
Location: perth australia
Posts: 1,287
Quote:
Originally Posted by rogerg View Post
Thanks for your offer Andrew. I have various objects which I have data like this for and some ongoing trying to get more .... so probably need to just work out a way to do it myself ... but might take you up on your offer once I have a complete set.
No problems Roger- you could always stick them on a USB drive and leave it up a Perth obs - you can drive pixinsight in batch mode, so it's just a question of processing time. Very little user time involved.
But as Rick has suggested - a weighted average is absolutely the only way to combine data with varying signal to noise.
Cheers
Andrew.
Reply With Quote
  #9  
Old 10-10-2013, 01:12 PM
rogerg's Avatar
rogerg (Roger)
Registered User

rogerg is offline
 
Join Date: Feb 2006
Location: Perth, Western Australia
Posts: 4,542
Quote:
Originally Posted by RickS View Post
I really like the ability to use a noise estimate for weighting, Roger. It makes mixing mismatched data work quite well. I have even integrated a bunch of narrowband and RGB master frames to make a "superluminance" for a narrowband/RGB hybrid image.

You probably have too much data for it to be convenient, but if you want me to try a noise weighted integration so you can see the result just drop me a PM. [Whoops, just saw Andrew's offer after I hit the button to post my reply...]

Good luck!

Cheers,
Rick.
I've found CCDStack can normalize and then stack the exposures weighted according to their exposure time. This sounds very similar except it's basing the weighting on exposure time rather than actual SNR. I suspect there would be some difference in the results from the two methods but stacking weighted by exposure might not end up so bad/different. I will give it a try.

I'm yet to be sure CCDStack doesn't have a similar "weight by noise" feature.

Quote:
Originally Posted by algwat View Post
Gidday ,

Try imageJ, create a stack of all the images.
use the drop menu selection image>>stacks>>Zproject.
Select either sum, average, medium, or maximum.
Adjust the contrast and brightness
drop menu selection image>>adjust>>bright-contrast.

Should be ok for Lum iimages, a bit tricky for RGB sets.

fin imageJ here, http://imagej.nih.gov/ij/index.html

kind regards, Alan
Thanks for your input Alan. I'm hesitant to get stuck in to yet another stacking program though! Learning PixInsight and now CCDStack has taken some time and brain power. I'll keep your idea of using imageJ in hand in case I refer back to it ...

Quote:
Originally Posted by alocky View Post
No problems Roger- you could always stick them on a USB drive and leave it up a Perth obs - you can drive pixinsight in batch mode, so it's just a question of processing time. Very little user time involved.
But as Rick has suggested - a weighted average is absolutely the only way to combine data with varying signal to noise.
Cheers
Andrew.
Andrew the data transfer isn't an issue, I upload 4-6GB every day anyhow so a couple'a hundred mb isnt' an issue. It's more about being able to do it myself anytime ongoing which I'm sure you realise ... Looking for "weighting" and ways to stack images "weighted" by different metrics has given me somewhere to head. I'll see how I go.
Reply With Quote
  #10  
Old 10-10-2013, 01:32 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 15,472
CCDstack will also do adaptive darks so if you have a bias frame then it will adjust each frame to a weight. You can see it doing that in the dialogue box.

The other way is simply process all the 300's to a master. Then process all the 180s to a master. Then combine the 2 with the various combine methods and pick the one you like the best.

Greg.
Reply With Quote
  #11  
Old 10-10-2013, 04:41 PM
naskies's Avatar
naskies (Dave)
Registered User

naskies is offline
 
Join Date: Jul 2011
Location: Brisbane
Posts: 1,865
Quote:
Originally Posted by rogerg View Post
I've found CCDStack can normalize and then stack the exposures weighted according to their exposure time. This sounds very similar except it's basing the weighting on exposure time rather than actual SNR. I suspect there would be some difference in the results from the two methods but stacking weighted by exposure might not end up so bad/different. I will give it a try.

I'm yet to be sure CCDStack doesn't have a similar "weight by noise" feature.
According to the CCDStack website, it does have this feature:

Quote:
Normalization:

Normalizing the stack results in all images having similar ADU values for corresponding pixels (and area and features). Normalization mathematically compensates for variations in sky background, sky transparency, exposure times and so on. Such compensation is often necessary to produce optimal data rejection and image combines.

For example, after normalization the average ADU of a 5-minute exposure will approximate the average ADU of a 10-minute exposure (thatís what normalization does). If these images are summed without weights then the 10-minute exposure will contribute the same S/N as the 5-minute exposure, thus resulting in sub-optimal S/N. But a weighted sum preserves this difference to produce optimal S/N.
Reply With Quote
  #12  
Old 10-10-2013, 04:57 PM
rogerg's Avatar
rogerg (Roger)
Registered User

rogerg is offline
 
Join Date: Feb 2006
Location: Perth, Western Australia
Posts: 4,542
Quote:
Originally Posted by naskies View Post
According to the CCDStack website, it does have this feature ...
Yeap, that was what I was interpreting as "normalize and then stack the exposures weighted according to their exposure time" which I had the impression would be different to what Rick said in PixInsight of .... "using normalization Additive with scaling and weighted by noise evaluation" ?

Perhaps they are the same but I interpreted that PixInsight bases the weighting on a measure of SNR and interpreted that CCDStack weights based on exposure time (it can't do it based on ADU because it's normalised them....) ?
Reply With Quote
  #13  
Old 10-10-2013, 05:02 PM
rogerg's Avatar
rogerg (Roger)
Registered User

rogerg is offline
 
Join Date: Feb 2006
Location: Perth, Western Australia
Posts: 4,542
Quote:
Originally Posted by gregbradley View Post
CCDstack will also do adaptive darks so if you have a bias frame then it will adjust each frame to a weight. You can see it doing that in the dialogue box.

The other way is simply process all the 300's to a master. Then process all the 180s to a master. Then combine the 2 with the various combine methods and pick the one you like the best.

Greg.
Darks - no problem, I haven't considered reduction in this thread because for the purposes of my stacking I'm considering that my images are perfectly reduced. In reality I perform reduction in Maxim before going to CCDStack at the moment.

So, when it comes to stacking I'm starting with 180s L reduced/calibrated frames and 300s L reduced/calibrated frames.

I agree all the 300's and all the 180's could be combined to two master files.

My real question lies in your simplistically put "combine the 2 with various combine methods" What I'm asking is what combine method I should use to give appropriate weighting. I think I have the answer ... weighted-sum in CCDStack, which might or might not be different to PixInsight's method Rick has mentioned
Reply With Quote
  #14  
Old 10-10-2013, 05:39 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,584
Quote:
Originally Posted by rogerg View Post
I think I have the answer ... weighted-sum in CCDStack, which might or might not be different to PixInsight's method Rick has mentioned
Yes, it is different. PixInsight's image normalization for combination does roughly the same thing as CCDStack's normalization - it attempts to deal with differences in mean background level and dispersion between subs. The noise estimation and consequent weighting done by PI is a separate function that doesn't appear to be done by CCDStack.

You may still get quite reasonable results just from "simple" normalization.

I like to squeeze the last drop of SNR out of my data so I tweak the algorithms and parameters and look at the SNR numbers as well as the rejection maps and the integration result. That probably comes from being a mildly OCD computer geek who is trapped in management

Cheers,
Rick.
Reply With Quote
  #15  
Old 10-10-2013, 06:21 PM
naskies's Avatar
naskies (Dave)
Registered User

naskies is offline
 
Join Date: Jul 2011
Location: Brisbane
Posts: 1,865
Quote:
Originally Posted by rogerg View Post
Yeap, that was what I was interpreting as "normalize and then stack the exposures weighted according to their exposure time" which I had the impression would be different to what Rick said in PixInsight of .... "using normalization Additive with scaling and weighted by noise evaluation" ?

Perhaps they are the same but I interpreted that PixInsight bases the weighting on a measure of SNR and interpreted that CCDStack weights based on exposure time (it can't do it based on ADU because it's normalised them....) ?
My apologies, I was going by your opening post. Normalizing for sky background means additive (moving the histogram left/right but keeping the shape the same), and normalizing for transparency means scaling (stretching the histogram peak so it's wider/narrower but keeping the mean/median ADU the same).

Yes, weighting by evaluated noise is something else again and will also compensate for other things like periods of poor transparency, or an unlucky number of cosmic ray/radiation hits.

PI has a range of options for weighting - based on noise has been giving me the best results so far.
Reply With Quote
  #16  
Old 10-10-2013, 08:52 PM
rogerg's Avatar
rogerg (Roger)
Registered User

rogerg is offline
 
Join Date: Feb 2006
Location: Perth, Western Australia
Posts: 4,542
Ok, have played around in CCDStack. See attached where the weight is shown and it's obviously taking exposure time in to account one way or another, as the 300s exposures are in the 4's and 180s exposures are in the 2's.

Considering my PC limitations at the moment which make PI a bit more problematic than it already would be on a good PC, I'll just trust CCDStack is doing the best I can for now.

Thanks all for your advice, I learned some useful info from this thread.
Attached Thumbnails
Click for full-size image (CCDStck diff L exposures.png)
54.2 KB9 views
Reply With Quote
  #17  
Old 10-10-2013, 09:05 PM
naskies's Avatar
naskies (Dave)
Registered User

naskies is offline
 
Join Date: Jul 2011
Location: Brisbane
Posts: 1,865
According to the manual, weights are calculated based on the inverse of the scale factor - i.e. it takes into account both the sub duration and sky transparency but not other sources of noise:

Quote:
Weights

Weights are calculated from normalization as the inverse of the scalar factor. The combine procedures use these weights to optimize the resulting S/N.
Changes in gain (due to different cameras in the same stack or variable gain due to binning) are factored into the weights so that all of the images are normalized on the electron level. The gain of the reference image (top image) is used to normalize the other gains (if different) and this affects the weights.
For example, after normalization the average ADU of a 5-minute exposure will approximate the average ADU of a 10-minute exposure (thatís what normalization does). If these images are summed without weights then the 10-minute exposure will contribute the same S/N as the 5-minute exposure, thus resulting in sub-optimal S/N. But a weighted sum preserves this difference to produce optimal S/N.

The weights are shown in image manager. It is possible to override a weight by entering a new weight in the image manager form. One possible reason to modify a weight might be to decrease the impact of images with poorer resolution.
http://www.ccdware.com/Files/ccdstack132.pdf
Reply With Quote
  #18  
Old 10-10-2013, 09:59 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 15,472
Yeah I was going to say that CCDstack must do that automatically as there is no combine option of weighted sum.

Median combine is the usual choice as it gets rid of outliers better than the other methods. Sum can be useful in some faint nebula type shots.

My go-to combine is median. If you have enough subs you can see the flaws disappear like satellites etc.

I also do normalising then hot and cold pixel rejection as a standard action also. It gets rid of the inevitable coloured dots that often appear without doing that step.

Greg.
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 09:38 PM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Advertisement
Meade Australia
Advertisement
Lunatico Astronomical
Advertisement
SkyWatcher Australia
Advertisement
Bintel
Advertisement
OzScopes Authorised Dealer
Advertisement
Celestron Australia
Advertisement
Astromechanics
Advertisement
Astronomy and Electronics Centre
Advertisement