#1  
Old 07-10-2014, 06:05 PM
PRejto's Avatar
PRejto (Peter)
Registered User

PRejto is offline
 
Join Date: Jan 2011
Location: Rylstone, NSW, Australia
Posts: 1,397
Stupid Question re Black Specks on FITS

This is probably pretty naive but I never really looked all that close at my raw fits files. Zoomed in a lot I see all kinds of black marks. These have become an issue when I try to set the final black point after processing, especially on photos where I have a narrow dynamic range due to light pollution.

The image is from a typical frame of my KAF8300 CCD.

Thanks!

Peter
Attached Thumbnails
Click for full-size image (00000014.PGC_564488.jpg)
163.6 KB57 views
Reply With Quote
  #2  
Old 07-10-2014, 07:54 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 17,877
That looks like you've zoomed in enormously, right?

Accurate dark subtracts with enough subs in the master dark at the same temp and exposure length should take care of noise. My FLI ML8300 at 1x1 10mins and -40C seems extremely clean. But if I zoom in 500% I see some black specs. They don't affect image quality.

Greg.
Attached Thumbnails
Click for full-size image (Dark Master bin1x1 -40C 600 seconds Scaled.jpg)
105.4 KB41 views

Last edited by gregbradley; 07-10-2014 at 08:34 PM.
Reply With Quote
  #3  
Old 07-10-2014, 07:59 PM
LewisM's Avatar
LewisM
Novichok test rabbit

LewisM is offline
 
Join Date: Aug 2012
Location: Somewhere in the cosmos...
Posts: 10,388
Quote:
Originally Posted by gregbradley View Post
That look like you've zoomed in enormously, right?

Accurate dark subtracts with enough subs in the master dark at the same temp and exposure length should take care of noise. My FLI ML8300 at 1x1 10mins and -40C seems extremely clean. But if I zoom in 500% I see some black specs. They don't affect image quality.

Greg.
I have the same CCD except OSC version, and I concur - exceedingly clean to the point that as yet I still have not applied flats, darks or bias frames.I dare say even better than any Sony chip I used.

I think these are just areas needing more photons Gaps if you will
Reply With Quote
  #4  
Old 07-10-2014, 08:07 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,584
Also hard to say much without knowing the screen stretch that has been applied. This can exaggerate tiny differences between the pixels. It's worth looking at the individual data values before and after calibration.

Cheers,
Rick.
Reply With Quote
  #5  
Old 07-10-2014, 09:07 PM
PRejto's Avatar
PRejto (Peter)
Registered User

PRejto is offline
 
Join Date: Jan 2011
Location: Rylstone, NSW, Australia
Posts: 1,397
Thanks for all the replies. Yes, the photo is the default stretch in CCDStack but magnified 400x. The only reason I'm looking at this is that stacks of luminance images taken with my KAF830 chip show a lot of these dark blotches. The individual subs also show the darkies but they seem to come to greater prominence after stacking and especially when stretching.

I've just gone back again and tried to see differences between uncalibrated and calibrated subs, and I've taken Rick's advice and looked at values before and after. I suspect something isn't going according to plan during calibration. The dark spots do not calibrate out.

Here are some typical numbers coming out of CCDStack (I selected the entire image);

Sub 480 seconds

No calibration:
mean 2,312.68
STD 598.94
S/N 3.86
median 2,298.00
int mode 2,297
min 915.00
max 65,535.00

Dark Only:

mean 1,835.27
STD 584.95
S/N 3.14
median 1,822.61
int mode 1,828
min -986.60
max 65,069.50

(Why is S/N getting worse? Why is min a neg number? Why median decreasing?)

Flat Only:

mean 2,181.57
STD 589.93
S/N 3.70
median 2,167.16
int mode 2,166
min 1,784.35
max 67,345.16

Dark + Flat:

mean 1,809.41
STD 577.36
S/N 3.13
median 1,796.61
int mode 1,792
min -1,009.29
max 66,947.62

It looks like calibration is only making things worse! Is that a correct interpretation?

I've taken 20 flats with a light box (and bias subtracted):

mean 33,300.01
STD 420.41
S/N 79.21
median 33,343.47
int mode 33,288
min 9,102.81
max 51,980.00

40 bias frames and 20 darks. I probably should take double the darks and perhaps triple the flats since I'm not dithering (yet,,,two scopes on one mount is a challenge!).

Much appreciated if someone here can shed light on this data.

Thanks,

Peter
Reply With Quote
  #6  
Old 07-10-2014, 09:39 PM
Shiraz's Avatar
Shiraz (Ray)
Registered User

Shiraz is offline
 
Join Date: Apr 2010
Location: ardrossan south australia
Posts: 4,918
Quote:
Originally Posted by PRejto View Post
Thanks for all the replies. Yes, the photo is the default stretch in CCDStack but magnified 400x. The only reason I'm looking at this is that stacks of luminance images taken with my KAF830 chip show a lot of these dark blotches. The individual subs also show the darkies but they seem to come to greater prominence after stacking and especially when stretching.

I've just gone back again and tried to see differences between uncalibrated and calibrated subs, and I've taken Rick's advice and looked at values before and after. I suspect something isn't going according to plan during calibration. The dark spots do not calibrate out.

Here are some typical numbers coming out of CCDStack (I selected the entire image);


Much appreciated if someone here can shed light on this data.

Thanks,

Peter
Hi Peter.

did a quick back of the envelope and your flat stack should have a SNR of about 70, so what you have looks reasonable.

The splotchiness looks to me like normal stretched random noise (mainly shot noise). It will not calibrate out on an individual sub because a sub is a lot noisier than a stack of calibration images - the noise will change a bit, but it will only be at the stage of integration that you will notice improvements from calibration. If you do not get an improvement at stacking, then you have fixed pattern noise in the flats - most likely they contain imprinted dark noise. You cannot get rid of fixed pattern noise by integrating more lights - the only solution (apart from dither) is to improve the calibration by having more flats and darks - as you suggest.

Whole image measures of noise etc. can be very misleading because they incorporate hot pixels and image structure. The most reliable way to measure image noise is to select a small patch of background sky with no stars or obvious hot pixels.
Reply With Quote
  #7  
Old 07-10-2014, 09:52 PM
rat156's Avatar
rat156
Registered User

rat156 is offline
 
Join Date: Aug 2005
Location: Melbourne
Posts: 1,694
Hi Peter,

Are you using adaptive dark frames? I use these of a very long DF exposure and get negative numbers for some of the hot pixels.

What software do you use to take your calibration frames?

Flats will not change the sign of the minimum value as the operator is a division.

Attached are two images from a stack of images of C/2013 V5 that I took last night, apart from being two different images, one has been calibrated and one not so, you can see the hot pixel to the left and a bit below the comet? Its value in the uncalibrated image is 65k, and -512 in the calibrated image.

Doesn't seem to matter in the final image though.

Cheers
Stuart
Attached Thumbnails
Click for full-size image (C2013V5Luminance00000142.jpg)
21.6 KB39 views
Click for full-size image (C2013V5Luminance00000138.jpg)
27.4 KB31 views
Reply With Quote
  #8  
Old 07-10-2014, 10:29 PM
PRejto's Avatar
PRejto (Peter)
Registered User

PRejto is offline
 
Join Date: Jan 2011
Location: Rylstone, NSW, Australia
Posts: 1,397
Quote:
Originally Posted by Shiraz View Post
Hi Peter.

did a quick back of the envelope and your flat stack should have a SNR of about 70, so what you have looks reasonable.

The splotchiness looks to me like normal stretched random noise (mainly shot noise). It will not calibrate out on an individual sub because a sub is a lot noisier than a stack of calibration images - the noise will change a bit, but it will only be at the stage of integration that you will notice improvements from calibration. If you do not get an improvement at stacking, then you have fixed pattern noise in the flats - most likely they contain imprinted dark noise. You cannot get rid of fixed pattern noise by integrating more lights - the only solution (apart from dither) is to improve the calibration by having more flats and darks - as you suggest.

Whole image measures of noise etc. can be very misleading because they incorporate hot pixels and image structure. The most reliable way to measure image noise is to select a small patch of background sky with no stars or obvious hot pixels.
Thanks very much for this Ray! I will be doing my flats over soon. I think the flat isn't working perfectly. It does remove the usual dust bunnies, but I'm noticing that the corners of subs are over corrected. I now wonder if I've got to get the light box closer to the objective because "something" is making the flat think it is darker in the corners than seems to be the case..

Peter
Reply With Quote
  #9  
Old 07-10-2014, 10:32 PM
PRejto's Avatar
PRejto (Peter)
Registered User

PRejto is offline
 
Join Date: Jan 2011
Location: Rylstone, NSW, Australia
Posts: 1,397
Quote:
Originally Posted by rat156 View Post
Hi Peter,

Are you using adaptive dark frames? I use these of a very long DF exposure and get negative numbers for some of the hot pixels.

What software do you use to take your calibration frames?

Flats will not change the sign of the minimum value as the operator is a division.

Attached are two images from a stack of images of C/2013 V5 that I took last night, apart from being two different images, one has been calibrated and one not so, you can see the hot pixel to the left and a bit below the comet? Its value in the uncalibrated image is 65k, and -512 in the calibrated image.

Doesn't seem to matter in the final image though.

Cheers
Stuart
Thanks Stuart! Yes, I did use an adaptive dark subtraction (12 in darks). In the past I've had better results not doing this. I appreciate your explanation for the negative values after calibraton.

Peter
Reply With Quote
  #10  
Old 16-10-2014, 12:23 PM
Paul Haese's Avatar
Paul Haese
Registered User

Paul Haese is offline
 
Join Date: Jan 2009
Location: Adelaide
Posts: 9,937
Just off the top of my head some things you may have over looked.

1 Are your darks current? ie less than 3 months.

2. in CCDstack check to see if you have time scaled checked on your calibration with regard to darks. ie if you have a 20 minute dark and applying that to a 10 minute light you can get dark blotches. Or if you have the correct length of both dark and light you might have the time scaling set it can cause this too. Easy to over look.

Also just check to see if data rejection set correctly. Under correction can cause stuff to slip through. Though in the case I don't think this is the problem.

Like others have said it is hard to determine what is going on from that cropped image and at what scale.
Reply With Quote
  #11  
Old 16-10-2014, 09:08 PM
PRejto's Avatar
PRejto (Peter)
Registered User

PRejto is offline
 
Join Date: Jan 2011
Location: Rylstone, NSW, Australia
Posts: 1,397
Hello Paul,

Thanks for your input here.

First, the darks were new (less than a few weeks) and I was adaptive subtracting. In my version of CCDStack I don't see anyplace to specify "time scaled" in the calibration setup. If I have a bias frame properly loaded along with a longer dark than the subframe, the calculation appears to proceed automatically. i.e., a 12 min dark is applied 50% to a 6 min sub. At least that is what the calibration info reports upon conclusion of calibration.

I think part of the issue might have been coming from flats that were not working properly due to a reflection. I'm at an impasse in testing further as I await an adapter that will allow me to move the imaging camera closer to the ONAG and still reach focus with the guiding camera. Hopefully this will solve 2 issues; vignetting and the reflection. After that I can see better how those dark things are doing.

Thanks again,

Peter
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 09:45 PM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Advertisement
Testar
Advertisement
Bintel
Advertisement