#1  
Old 26-09-2015, 08:40 AM
codemonkey's Avatar
codemonkey (Lee)
Lee "Wormsy" Borsboom

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 2,058
Snr

I did a brief experiment last night, after seeing some threads on CN lately where people are taking huge amounts (thousands) of very short (<= 4secs) subs and integrating them with seemingly good results; and I'm talking DSOs, not planetary. This had me questioning what I believed about SNR and image acquisition.

In the end, my brief experiment (no doubt flawed in many ways!) showed the conventional wisdom to be true: longer subs will give you better SNR, at least when you're "read noise limited".

All three of the attached images show a tiny crop of the edge of the Helix nebula. All images were calibrated. All images were stretched using PI's STF tool in combination with HT.

Where stacking was involved, Windsorized Sigma Clipping was used. Maybe not appropriate for the 5x1, but eh.

One is a single sub, 5mins in length.
One is 5 sub, 1min in length.
One is 57 subs, 1min in length.

Of the three images, one has a SNR of 4.8:1, another has a SNR of 5.5:1, the other has a SNR of 3.5:1

Can you guess which SNR belongs to which image?
Attached Thumbnails
Click for full-size image (snr_01.jpg)
17.8 KB88 views
Click for full-size image (snr_02.jpg)
22.9 KB73 views
Click for full-size image (snr_03.jpg)
21.9 KB66 views
Reply With Quote
  #2  
Old 26-09-2015, 09:27 AM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,584
It's more than conventional wisdom, Lee. It's basic physics and mathematics. As they say, Science works, *****es

I can easily pick the lowest SNR image. I'd be more confident of differentiating 2 and 3 if you hadn't changed the image scale. Was that meant to be a trick question?
Reply With Quote
  #3  
Old 26-09-2015, 09:38 AM
codemonkey's Avatar
codemonkey (Lee)
Lee "Wormsy" Borsboom

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 2,058
There's no change in image scale as such, but the previews aren't quite the same since I roughly created them manually. Apologies for the inconsistency. Have a crack anyway.

As you say, it's all physics and math and science works. Trouble is, however, a lot of people just parrot information that they heard/read and apply it to situations where it no longer holds. There's some questions I had that I hadn't seen satisfactory answers to and since it's easier for a lay person like me to just do it and see the results for myself, I'm happy to do just that.

Back on topic...which is which?
Reply With Quote
  #4  
Old 26-09-2015, 09:44 AM
peter_4059's Avatar
peter_4059 (Peter)
Big Scopes are Cool

peter_4059 is offline
 
Join Date: Jun 2007
Location: SE Tasmania
Posts: 4,532
There's a good Craig Stark read on SNR (4 parts) here:

http://www.cloudynights.com/page/art...t-part-1-r1895

http://www.cloudynights.com/page/art...ne-pixel-r1902

http://www.cloudynights.com/page/art...r-camera-r1929

http://www.cloudynights.com/page/art...sampling-r1970
Reply With Quote
  #5  
Old 26-09-2015, 09:50 AM
codemonkey's Avatar
codemonkey (Lee)
Lee "Wormsy" Borsboom

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 2,058
Thanks Peter :-)

Does anyone want to play? Maybe everyone already knows this, but I think it's an interesting topic and if it helps shed some light on things for just one person I'll be happy.
Reply With Quote
  #6  
Old 26-09-2015, 10:05 AM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,584
Quote:
Originally Posted by codemonkey View Post
There's no change in image scale as such, but the previews aren't quite the same since I roughly created them manually. Apologies for the inconsistency. Have a crack anyway.
1, 3, 2 from best to worst.
Reply With Quote
  #7  
Old 26-09-2015, 10:08 AM
codemonkey's Avatar
codemonkey (Lee)
Lee "Wormsy" Borsboom

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 2,058
Quote:
Originally Posted by RickS View Post
1, 3, 2 from best to worst.
Afraid not. That's what I would have thought just looking at them, and this is why I posted this thread.

For the record, I measured the ratio the same way that Craig Stark did in the articles Peter mentioned; stddev vs mean.
Reply With Quote
  #8  
Old 26-09-2015, 10:23 AM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,584
Quote:
Originally Posted by codemonkey View Post
Afraid not. That's what I would have thought just looking at them, and this is why I posted this thread.

For the record, I measured the ratio the same way that Craig Stark did in the articles Peter mentioned; stddev vs mean.
Is that based on a comparison of the mismatched previews?
Reply With Quote
  #9  
Old 26-09-2015, 10:35 AM
codemonkey's Avatar
codemonkey (Lee)
Lee "Wormsy" Borsboom

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 2,058
Quote:
Originally Posted by RickS View Post
Is that based on a comparison of the mismatched previews?
I doubt a couple of pixels is going to make that much of a difference, and I did it twice with different previews and got roughly the same numbers.

Let's stop playing guessing games now.

#1 whilst clearly the cleanest of the images, is the 57x1min exposure and has the 4.8:1 SNR - this is our midrange image in terms of SNR.

#2 has the worst SNR; it's the 5x1min sub integration.

#3 our SNR winner, is the 1x5min sub.

So "conventional wisdom" holds: if you're read noise limited, longer subs are better in terms of SNR.

But looking at these images, I question why we optimise for SNR? Honestly, which of these would you prefer to process? Without doubt it's #1.

Obviously #1 has more than 10 times the integration time than the others, but that's not the point. The point is I think we should be optimising for absolute noise, rather than SNR.
Reply With Quote
  #10  
Old 26-09-2015, 10:43 AM
peter_4059's Avatar
peter_4059 (Peter)
Big Scopes are Cool

peter_4059 is offline
 
Join Date: Jun 2007
Location: SE Tasmania
Posts: 4,532
I would have thought you'd do a comparison based on constant total integration time. ie if you have 3 hours total on a target how is your time best spent in terms of number of subs vs sub exposure length. Here's how Craig's calcs look for my SN10/QSI at Duckadang vs home (albeit on different targets). This pretty clearly shows for a constant total duration it is better to use a longer exposure to a point where you then get diminishing returns.
Attached Thumbnails
Click for full-size image (Duckadang_calc.JPG)
34.9 KB39 views
Click for full-size image (Home_calc.JPG)
34.4 KB30 views
Reply With Quote
  #11  
Old 26-09-2015, 10:48 AM
codemonkey's Avatar
codemonkey (Lee)
Lee "Wormsy" Borsboom

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 2,058
Thanks Peter, nice post! And yes, that's exactly what I had in mind. Unfortunately all of my 5min subs except one got ruined. So I compared 5x1 to 1x5 and then did the 57x1 just for giggles and then I saw something I didn't expect and I thought I'd share that.

My point still remains though: after seeing the images I captured here, do you not even start to question why we optimise for SNR? There's no doubt that when read noise limited longer subs will give you better SNR. But again, looking at those images, do you really care which one has the better SNR? If you did, you'd discard the cleanest of the images due to its inferior SNR.

So while you've measured the relative SNR, I'm not convinced that SNR in itself means much at all in practical application.
Reply With Quote
  #12  
Old 26-09-2015, 10:52 AM
peter_4059's Avatar
peter_4059 (Peter)
Big Scopes are Cool

peter_4059 is offline
 
Join Date: Jun 2007
Location: SE Tasmania
Posts: 4,532
I think you need to compare 50x1 min subs to 10x5 min and see which one looks cleaner. I suspect the 10x5 stack is going to have the higher SNR and look cleaner.
Reply With Quote
  #13  
Old 26-09-2015, 11:09 AM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,584
Quote:
Originally Posted by codemonkey View Post
My point still remains though: after seeing the images I captured here, do you not even start to question why we optimise for SNR? There's no doubt that when read noise limited longer subs will give you better SNR. But again, looking at those images, do you really care which one has the better SNR? If you did, you'd discard the cleanest of the images due to its inferior SNR.

So while you've measured the relative SNR, I'm not convinced that SNR in itself means much at all in practical application.
My experience is that things work in practice pretty much as expected in theory and that higher SNR images look better. Not sure what has happened with your experiment but I wouldn't jump to any conclusions

Noise is random and hard to estimate accurately unless you have a statistically significant number of samples.

Cheers,
Rick.
Reply With Quote
  #14  
Old 26-09-2015, 12:40 PM
Slawomir's Avatar
Slawomir (Suavi)
Registered User

Slawomir is offline
 
Join Date: Sep 2014
Location: North Queensland
Posts: 3,240
Hi Lee,

STF will stretch different data differently, and from what you wrote I gather you just applied STF 3 times for each preview, am I right?

This is what I would do to visually compare noise in different images: Apply STF to one of the previews, then copy that to histogram transformation tool, and then apply that histogram transformation to the remaining previews - in this way you are applying the same stretch for all 3 previews.

EDIT: I think perhaps it would be interesting to compare regions with stronger signal in these images, just for fun

EDIT2: Another random thought...it does not seem right just to look at background and on that basis decide potentially most effective integration. Taking that approach to the extreme...500 frames with millisecond exposure will generate even cleaner "background", but there will be nearly zero signal from a DSO. That's why my suggestion to compare areas with stronger signal as well

Last edited by Slawomir; 26-09-2015 at 01:00 PM.
Reply With Quote
  #15  
Old 26-09-2015, 02:09 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by RickS View Post
My experience is that things work in practice pretty much as expected in theory and that higher SNR images look better. Not sure what has happened with your experiment but I wouldn't jump to any conclusions

Noise is random and hard to estimate accurately unless you have a statistically significant number of samples.

Cheers,
Rick.
+1 - SNR really is the only measure that matters.

a couple of points:
you have used an averaging stack, which means that the signal will be proportional to the sub length and will not change as you add more subs. However, it will be 5x higher for the 5 min sub. To get a reasonable idea of how much noise you really have, you need to stretch differently so that the signal in each image is the same.
how have you measured SNR? you need a totally flat region to get an SD measure that incorporates just the noise and not some variation in the background level as well. FWIW, the method used in Nebulosity is very reliable.
Reply With Quote
  #16  
Old 26-09-2015, 02:18 PM
codemonkey's Avatar
codemonkey (Lee)
Lee "Wormsy" Borsboom

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 2,058
Quote:
Originally Posted by peter_4059 View Post
I think you need to compare 50x1 min subs to 10x5 min and see which one looks cleaner. I suspect the 10x5 stack is going to have the higher SNR and look cleaner.
Yeah, that's the plan. Actually, the original plan was to take a crazy amount of very short subs (~5s) and compare, but it becomes very difficult to register narrowband subs of that length.

Quote:
Originally Posted by RickS View Post
My experience is that things work in practice pretty much as expected in theory and that higher SNR images look better. Not sure what has happened with your experiment but I wouldn't jump to any conclusions

Noise is random and hard to estimate accurately unless you have a statistically significant number of samples.

Cheers,
Rick.
Fair enough, thanks for your thoughts, Rick.

Quote:
Originally Posted by Slawomir View Post
Hi Lee,

STF will stretch different data differently, and from what you wrote I gather you just applied STF 3 times for each preview, am I right?

This is what I would do to visually compare noise in different images: Apply STF to one of the previews, then copy that to histogram transformation tool, and then apply that histogram transformation to the remaining previews - in this way you are applying the same stretch for all 3 previews.

EDIT: I think perhaps it would be interesting to compare regions with stronger signal in these images, just for fun

EDIT2: Another random thought...it does not seem right just to look at background and on that basis decide potentially most effective integration. Taking that approach to the extreme...500 frames with millisecond exposure will generate even cleaner "background", but there will be nearly zero signal from a DSO. That's why my suggestion to compare areas with stronger signal as well
Hey S :-)

That's an interesting point about the presentation/stretch. If I do the same stretch for all of them one is going to be much brighter than the others (because it's exposure was 5 times longer), which I think might make it more difficult to compare noise. Having said that, a non-linear transformation probably isn't the best choice for presenting this either because it could have impacted the comparison.

This isn't background by the way, it's the edge of the brighter region of the nebula; it captures probably the strongest area of signal in the image and where it falls off a bit.
Reply With Quote
  #17  
Old 26-09-2015, 02:46 PM
Slawomir's Avatar
Slawomir (Suavi)
Registered User

Slawomir is offline
 
Join Date: Sep 2014
Location: North Queensland
Posts: 3,240
Quote:
Originally Posted by codemonkey View Post
Hey S :-)

That's an interesting point about the presentation/stretch. If I do the same stretch for all of them one is going to be much brighter than the others (because it's exposure was 5 times longer), which I think might make it more difficult to compare noise. Having said that, a non-linear transformation probably isn't the best choice for presenting this either because it could have impacted the comparison.

This isn't background by the way, it's the edge of the brighter region of the nebula; it captures probably the strongest area of signal in the image and where it falls off a bit.
Lee

Maybe linear fit applied before a stretch? Just for fun
Reply With Quote
  #18  
Old 26-09-2015, 04:08 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
maybe a silly question, but assume that you measured SNR on the unstretched data?

with your system, the turnover point on Peter's curves will be at about 5 minute lum subs in reasonably dark sky. For RGB, about 10 minutes.
Reply With Quote
  #19  
Old 26-09-2015, 04:42 PM
codemonkey's Avatar
codemonkey (Lee)
Lee "Wormsy" Borsboom

codemonkey is offline
 
Join Date: Jul 2013
Location: Kilcoy, QLD
Posts: 2,058
Quote:
Originally Posted by Slawomir View Post
Lee

Maybe linear fit applied before a stretch? Just for fun
I'll have a play later and see what I can come up with :-)

Quote:
Originally Posted by Shiraz View Post
maybe a silly question, but assume that you measured SNR on the unstretched data?

with your system, the turnover point on Peter's curves will be at about 5 minute lum subs in reasonably dark sky. For RGB, about 10 minutes.
Yep, SNR was measured on unstretched data.

I found that of the 5min subs I mentioned before, only two of them were ruined by clouds, the others just had tracking/guiding issues, which means they're usable for SNR comparisons.

I combined 40x1min and 8x5min and found that the former had better SNR. That's not what I expected to see either. Not sure which looked better, I had difficulty eyeballing them. Might post up some samples a bit later.

I'd encourage anyone interested to have a look around on CN. There's people posting samples of DSOs taken with subs as short as 0.1s, just using heaps (thousands) of subs. This is what caused me to start this experiment.
Reply With Quote
  #20  
Old 26-09-2015, 05:12 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,584
Quote:
Originally Posted by codemonkey View Post
I'd encourage anyone interested to have a look around on CN. There's people posting samples of DSOs taken with subs as short as 0.1s, just using heaps (thousands) of subs. This is what caused me to start this experiment.
There are a few folks doing short subs with low read noise cameras. That's perfectly fine because their subs, though short, aren't read noise limited. One day we will all probably be doing lucky imaging of DSOs with very short subs but we'll need affordable very low read noise sensors first.

Anybody doing very short subs with the sort of cameras that most of us currently use is getting suboptimal results. That doesn't mean that they aren't getting OK results. It does mean that they could get the same results in less time with longer subs.

This is not just an opinion. It's the way that photons and sensors work.

By all means keep experimenting, Lee. I'm sure you'll learn a lot from it and hopefully others will too. If nothing else, you'll start some more lively discussions

Cheers,
Rick.
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 03:20 AM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Advertisement
Testar
Advertisement
Bintel
Advertisement