#1  
Old 09-09-2017, 06:26 PM
DiscoDuck's Avatar
DiscoDuck (Paul)
Raider Nation

DiscoDuck is offline
 
Join Date: Apr 2014
Location: Adelaide
Posts: 691
ASI1600MM-C noise

A question about the ASI1600MM-C's noise. Attached is a blown up (3:1) crop of a highly stretched single sub of 30s duration (in 47 Tuc). There are numerous black dots in there, typically in the lower signal areas.

Though they pretty much disappear in cosmetic correction and stacking in PI, I'm trying to understand if they are an issue. They are not dead pixels, as they change location from frame to frame. They seem to be very big outliers in the distribution though, as the overall noise level is relatively low.

A similar (probably same) effect I've noticed with biases e.g. mean ADU value is say at 300 with a std dev of about 30 say, and yet there are many points with values down at 16. Seems like there's a very long tail on the noise distribution. The rejection maps in Image Integration in PI have a lot of data in them, whereas for my old QHY8 they were essentially black.

Is this normal for this camera? Or has mine got some issues?

Comments appreciated.

Paul

PS If it makes any difference, this is at unity gain (and offset the default of 21) and -15C cooler temp.
Attached Thumbnails
Click for full-size image (crop.jpg)
86.4 KB42 views
Reply With Quote
  #2  
Old 09-09-2017, 08:13 PM
Atmos's Avatar
Atmos (Colin)
Ultimate Noob

Atmos is offline
 
Join Date: Aug 2011
Location: Melbourne
Posts: 6,980
They may just be cold pixels as opposed to hot pixels
Reply With Quote
  #3  
Old 09-09-2017, 09:00 PM
DiscoDuck's Avatar
DiscoDuck (Paul)
Raider Nation

DiscoDuck is offline
 
Join Date: Apr 2014
Location: Adelaide
Posts: 691
Quote:
Originally Posted by Atmos View Post
They may just be cold pixels as opposed to hot pixels
It was a very chilly night!
As I said though, different pixels impacted from one sub to the next.
Reply With Quote
  #4  
Old 09-09-2017, 09:08 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Probably normal Paul.

The sub signal can only have values in steps of 16, so eg in your bias, the values can only be .....288,304,320,336.... etc. At gain139, these 16ADU steps are equivalent to single electron changes in the signal, so there is not any point in getting finer resolution. Shot and read noise, will show as a random smattering of noticeably lighter and darker pixels - all of the noise affected pixels can have only a limited number of values about the signal mean, so you see a few distinctly lighter and darker pixels and nothing with in-between values.

Stacking enough subs fills in the intermediate values in the final image, but the data distribution in the individual subs is very sparse at high gain (eg 139). Of course that means that standard non-linear rejection methods have to be used carefully. Sigma rejection in particular can throw away lots of data and leave distinct holes in the resulting stack histogram by rejecting a quite large number of pixels that have the same value (eg, if you ask it to reject 5% outliers, it may not be able to do so if the steps in the sub exposure histograms only give the option of rejecting say 1%,3% or 10% - it will reject 10%, which may make a noticeable difference to the data). I have found that min/max rejection works best on dithered short subs - throwing away a fixed percentage of the data at the top and bottom gets rid of almost all cosmic ray events and most hot pixel leakage without disturbing the real data distribution too much.

Interesting chip and, at high gain, completely different to a normal CCD in many ways. cheers Ray

Last edited by Shiraz; 09-09-2017 at 09:33 PM.
Reply With Quote
  #5  
Old 09-09-2017, 09:28 PM
DiscoDuck's Avatar
DiscoDuck (Paul)
Raider Nation

DiscoDuck is offline
 
Join Date: Apr 2014
Location: Adelaide
Posts: 691
Quote:
Originally Posted by Shiraz View Post
Normal Paul.

The signal can only take values in steps of 16, so eg in the bias, the values can only be .....304,320,336.... etc. At 139, these steps are equivalent to single electron changes in the signal, so there is not any point in getting finer resolution. With shot and read noise, this will show as a smattering of noticeably lighter and darker pixels - all of which can have only a limited number of values.
Thanks Ray. Understood. But that is a lot of points at 10 sigma from the mean I'm seeing I guess. Still - happy to hear it's normal!

Quote:
Originally Posted by Shiraz View Post
Stacking fills in the intermediate values in the final image, but the data distribution in the subs is very ragged. Of course that means that standard non-linear rejection methods have to be used carefully. Sigma rejection in particular can throw away lots of data and leave distinct holes in the resulting stack histogram by rejecting a quite large number of pixels that have the same value (eg, if you ask it to reject 5% outliers, it may not be able to do so if the steps in the histogram only give the option of rejecting say 1%,3% or 10% - it will reject 10% and make a noticeable difference to the data). I have found that min/max rejection works best on dithered short subs - throwing away a fixed percentage of the data at the top and bottom gets rid of almost all cosmic ray events and most hot pixel leakage without disturbing the real data distribution too much.
So that would be for each integration, i.e. bias, dark, flat, light? And normalisation options as per normal on them? Min/Max rejection takes as parameters the numbers of pixels to reject it seems (just looking now) - what sort of values do you use in your case?
Reply With Quote
  #6  
Old 09-09-2017, 09:48 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
well, maybe I was a bit premature on that advice - apologies.
just had a look at a bias sub from mine - it has a smattering of dark pixels, but they are not that far from the mean. A typical 21x21 pixel region has a mean of ~290, min of 224, max 352 (bias21, gain 100). That is a range of about 3 sigma, with nothing at very low values, so your camera is behaving somewhat differently. I seem to recall that these cameras can do strange things with very short bias subs - could that be an issue? I went over to 1 second bias subs early on, but do not have any older very short subs to have a look at.

yep, min/max for all stacks seems to work best. For lights, I have been using sparse dither where dither only applies every 3 frames, and have been using rejection of the 4 highest and 4 lowest (all standard normalisation) - seems to do fairly well. not perfect, but the occasional leaked hot pixel is fairly easy to deal with. cheers Ray

edit: just another thought, with short exposures there is so little signal that it is often necessary to add in a pedestal when calibrating to ensure that the data cannot try to go negative when the dark+bias is subtracted. Might be unrelated to your issue, but it can cause odd data distributions.

Last edited by Shiraz; 09-09-2017 at 10:27 PM.
Reply With Quote
  #7  
Old 09-09-2017, 10:08 PM
billdan's Avatar
billdan (Bill)
Registered User

billdan is offline
 
Join Date: Mar 2012
Location: Narangba, SE QLD
Posts: 1,551
Over at Cloudy Nights someone had the same problem with Ha subs.

One suggestion to try, was to increase the offset to stop black clipping.

Another replied to increase the USB speed limit from default 40 to 80.

However the OP never came back, so we don't if any of these suggestions worked.
Reply With Quote
  #8  
Old 09-09-2017, 10:52 PM
glend (Glen)
Registered User

glend is offline
 
Join Date: Jun 2013
Location: Lake Macquarie
Posts: 7,033
Quote:
Originally Posted by Shiraz View Post
well, maybe I was a bit premature on that advice - apologies.
just had a look at a bias sub from mine - it has a smattering of dark pixels, but they are not that far from the mean. A typical 21x21 pixel region has a mean of ~290, min of 224, max 352 (bias21, gain 100). That is a range of about 3 sigma, with nothing at very low values, so your camera is behaving somewhat differently. I seem to recall that these cameras can do strange things with very short bias subs - could that be an issue? I went over to 1 second bias subs early on, but do not have any older very short subs to have a look at.

yep, min/max for all stacks seems to work best. For lights, I have been using sparse dither where dither only applies every 3 frames, and have been using rejection of the 4 highest and 4 lowest (all standard normalisation) - seems to do fairly well. not perfect, but the occasional leaked hot pixel is fairly easy to deal with. cheers Ray

edit: just another thought, with short exposures there is so little signal that it is often necessary to add in a pedestal when calibrating to ensure that the data cannot try to go negative when the dark+bias is subtracted. Might be unrelated to your issue, but it can cause odd data distributions.
Interesting Ray, i have always used 0 sec bias frames when using SGP, because it can. Sounds like i should be using 1 sec bias. My whole 1600 bias library is built on 0 sec subs.
Reply With Quote
  #9  
Old 09-09-2017, 11:20 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by glend View Post
Interesting Ray, i have always used 0 sec bias frames when using SGP, because it can. Sounds like i should be using 1 sec bias. My whole 1600 bias library is built on 0 sec subs.
Hi Glen. My camera certainly had early problems with very short bias subs and I have stuck with 1 second ones ever since - however, can't find any early bias data to see what the exact problem was. In all likelihood, later software may well have fixed the issue - whatever it was - but I don't like to meddle with stuff that is working, so my bias subs remain at 1 second. Just raised sub length as a possibility that Paul might like to look at for his camera. Do you have any obvious anomalous values in your short bias subs? - your images would suggest that your bias library is fine. Cheers Ray

Last edited by Shiraz; 09-09-2017 at 11:35 PM.
Reply With Quote
  #10  
Old 10-09-2017, 09:20 AM
DiscoDuck's Avatar
DiscoDuck (Paul)
Raider Nation

DiscoDuck is offline
 
Join Date: Apr 2014
Location: Adelaide
Posts: 691
Thanks Ray and Glen.

I'll do some experiments with offset, USB speed and longer bias frames.

Your advice much appreciated.
Reply With Quote
  #11  
Old 10-09-2017, 09:31 AM
Merlin66's Avatar
Merlin66 (Ken)
Registered User

Merlin66 is offline
 
Join Date: Oct 2005
Location: Junortoun Vic
Posts: 8,904
Guys,
You may be interested in the data being collected by Christian Buil.
http://www.astrosurf.com/buil/CMOSvsCCD/index.html

There's also some healthy discussion on the astronomical spectroscopy group:
https://groups.yahoo.com/neo/groups/...messages/13841
Reply With Quote
  #12  
Old 10-09-2017, 04:48 PM
Camelopardalis's Avatar
Camelopardalis (Dunk)
Drifting from the pole

Camelopardalis is offline
 
Join Date: Feb 2013
Location: Brisbane
Posts: 5,425
I don't use bias frames but use a dark library of ~100 frames per master.

You should check your bias value...I found mine to require 37 to bring the curve off the left edge of the histogram at unity gain.
Reply With Quote
  #13  
Old 10-09-2017, 07:19 PM
DiscoDuck's Avatar
DiscoDuck (Paul)
Raider Nation

DiscoDuck is offline
 
Join Date: Apr 2014
Location: Adelaide
Posts: 691
Thanks Ken and Dunk.
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 08:46 PM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Advertisement
Bintel
Advertisement
Testar
Advertisement