Go Back   IceInSpace > Equipment > Astrophotography and Imaging Equipment and Discussions

Reply
 
Thread Tools Rating: Thread Rating: 7 votes, 5.00 average.
  #1  
Old 28-09-2013, 12:47 AM
naskies's Avatar
naskies (Dave)
Registered User

naskies is offline
 
Join Date: Jul 2011
Location: Brisbane
Posts: 1,865
Overscan calibration to improve dark frame scaling (SBIG STF-8300M + PI)

Before I begin... thanks must go to RickS and the authors of various astro texts/webpages for informing me about overscan calibration and for helping me to get it to work in PI

Background: most of us are familiar with bias, dark, light, and flat frames. With bias frames, we can use scaled darks - i.e. take a single set of dark frames say 10 min in duration, and mathematically scale them for calibration against other sub durations (5, 10, 15, 20, 30 min etc).

The problem is that when you do the pixel math calculations by hand, you may find that it doesn't actually work. For example, a 20 min master dark frame might have only 150% of the signal in a 10 min master dark frame - rather than the 200% you'd expect according to theory.

It turns out there are two separate components to the bias signal: a fixed column-by-column (or row-by-row) pattern, PLUS an offset that affects every pixel equally but which changes with every frame. Master bias frames calibrate for the former, but for the latter we need overscan calibration.

Overscan calibration: The overscan region is a non-imaging part of the CCD sensor used purely for calibration. That is, these pixels don't receive light (they may be physically opaque up on the sensor), but they are still connected to the same circuitry for reading pixel data. Think of them as pixels that are permanently wired to take bias and dark frames (even when the rest of the CCD is actually taking a light frame).

It seems that until recently, most amateur astro imagers and camera manufacturers just ignored overscan pixels. Depending upon the camera manufacturer, they can be downloaded as extra rows and columns in an image with a bit of driver tweaking*.

The median value of the overscan pixels is then subtracted from all frames - bias, dark, flat, and light frames (so you need data reduction/processing software such as PixInSight that supports overscan to do this).

I've done some number crunching with dark frames on my SBIG STF-8300M operating at -10 deg C (see the table in the attached images). For simplicity's sake, just look at the "600 sec --> 3600 sec" row, which shows a mean ADU/pixel error of 199.9 and 32.8 for standard and overscan calibration, respectively. The 199.9 figure means that if I just use standard bias / dark / flat / light frame processing and scale a 10 min master dark up to 60 mins, the scaled dark will overshoot a 60 min master dark by 199.9 ADU/pixel on average. Using overscan calibration, the error is only 32.8 ADU/pixel - a 83.6% reduction!

Note: these numbers are based on the same image pixels - the only difference is whether the median overscan offset value is subtracted from each frame. Furthermore, overscan calibration would also improve the accuracy of flat frames.

Step-by-step: Here are the precise steps that I follow for an SBIG STF-8300M:

1. Upgrade to the latest camera firmware (mid-2013) - earlier versions of the firmware had bugs that resulted in grossly incorrect bias frames on my camera. Note, this will invalidate your bias/dark library.

2. Enable the overscan region in the SBIG 64 Bit Driver Checker Utility by clicking on the "Config Drvr" button and enter 60 into the Overscan Rows/Columns field. Turn off Auto Bias Level Correction*.

3. Launch MaximDL - it now detects the camera as 1706x1296 (instead of 1676x1266 using 2x2 binning). See the attached image - it adds dark bars to the right/bottom of the image. Capture bias + dark + light frames normally.

4. When reducing data in PixInSight, enable the Overscan function (see attached image) for all frames - bias, dark, flat and light. This crops the image back down to the original 1676x1266 pixels, and subtracts the median ADU from a 1676x24 pixel overscan region below the image. You may also need to set the Output pedestal to prevent negative values (I set it to 200 in this case).

5. You can integrate the resulting subs as per normal. I generated the numbers in the first table using the Statistics process in PI using the same subs, with and without overscan calibration.

Conclusion: The difference with overscan is not huge - you probably won't notice it in the final stacked output unless you're going really deep with the processing. That said, the KAF-8300M has a read noise a bit over 8e- (22 ADU) so the improvement here is significant. Since this reduces noise for "free" (takes no extra imaging time, equipment, or $$$), I think it's worth doing

I'm more than happy to clarify anything that I've explained poorly


* I believe some manufacturers (QHY maybe?) have added overscan calibration functionality directly into the driver so that you don't have to do it yourself. I don't know any specific details.

** The Auto Bias Level Correction function - which I assume is meant to be automatically applying overscan calibration - on my SBIG STF-8300M has never worked correctly and gives erratic results, despite numerous firmware updates.
Attached Thumbnails
Click for full-size image (overscan_cal_results_-10degC.png)
105.9 KB279 views
Click for full-size image (enabling_overscan_sbig.png)
21.0 KB213 views
Click for full-size image (raw_sub_with_overscan.jpg)
136.8 KB241 views
Click for full-size image (pixinsight_overscan_calibration.png)
139.7 KB246 views
Reply With Quote
  #2  
Old 28-09-2013, 11:55 AM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 15,044
Great post, thanks Dave.

Greg.
Reply With Quote
  #3  
Old 28-09-2013, 01:14 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,581
Nicely explained, Dave!

I was poking around my narrowband Helix data today and noticed that some of the faintest nebulosity that I'm able to see clearly in the stacked images corresponds to only 25e- in a 30 minute exposure. When you compare that to the 40e- bias drift I have seen from the camera I think it makes a convincing case for overscan calibration, especially when working with dim targets.

Cheers,
Rick.
Reply With Quote
  #4  
Old 28-09-2013, 01:56 PM
naskies's Avatar
naskies (Dave)
Registered User

naskies is offline
 
Join Date: Jul 2011
Location: Brisbane
Posts: 1,865
Thanks, gents. Yep, I agree for deep narrowband images it definitely seems worthwhile. For light pollution limited LRGB subs, it's probably not worth the trouble unless you're already doing it for NB.

I forgot to mention above - bottom line, with the 8300 chip (and presumably others too) I'd recommend taking dark frames at the longest duration you image at and scaling down for best results. For example, take a library of 30 min darks and scale them for LRGB subs at 5, 10, 15 mins etc. Doing it the other way (scaling up) result in much more error.
Reply With Quote
  #5  
Old 28-09-2013, 03:16 PM
RobF's Avatar
RobF (Rob)
Mostly harmless...

RobF is offline
 
Join Date: Jul 2008
Location: Brisbane, Australia
Posts: 5,399
I salute you Dave!
Reply With Quote
  #6  
Old 28-09-2013, 09:47 PM
troypiggo's Avatar
troypiggo (Troy)
Bust Duster

troypiggo is offline
 
Join Date: Nov 2008
Location: Brisbane, Australia
Posts: 4,790
Interesting. I'll need to go back and read this in more detail later.

Do you need to completely update your bias and dark libraries? Or does the clipping off of the overscan region take care of that?
Reply With Quote
  #7  
Old 28-09-2013, 10:28 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,581
G'day Troy,

You need to collect a new set of bias and dark frames with the overscan attached and then build new master calibration frames.

Cheers,
Rick.
Reply With Quote
  #8  
Old 01-10-2013, 02:30 PM
Bart's Avatar
Bart
Don't have a cow, Man!

Bart is offline
 
Join Date: Apr 2013
Location: Adelaide, South Australia
Posts: 657
In image 4 of the OP, on the line that says Source Region in the Overscan section of the Image Calibration tool, what is the 24 and how is this derived and what does it mean? Could you in general explain that section a bit more for me as my camera has an overscan option and am interested to give it a go.

Thanks.
Reply With Quote
  #9  
Old 01-10-2013, 03:13 PM
naskies's Avatar
naskies (Dave)
Registered User

naskies is offline
 
Join Date: Jul 2011
Location: Brisbane
Posts: 1,865
Quote:
Originally Posted by Bart View Post
In image 4 of the OP, on the line that says Source Region in the Overscan section of the Image Calibration tool, what is the 24 and how is this derived and what does it mean? Could you in general explain that section a bit more for me as my camera has an overscan option and am interested to give it a go.

Thanks.
There are several types of overscan pixels, but the ones I was interested in were (according to the KAF-8300M datasheet):

* Dark Dummy Pixels - working pixels that are never exposed to light; it's like normal pixels from a Dark frame but available in every image, and

* Virtual Dummy Pixels - simulated pixels that aren't connected to the vertical shift register (so bias + read noise is present, but no thermal noise or light data); which is like Bias frame pixels but available in every image.

The overscan data download in the SBIG driver in no way correlated with the chip's pixel overscan geometry, i.e. I couldn't match specific overscan pixels to Dark/Virtual Dummy Pixels in the datasheet's diagram (neither could the person I emailed to at SBIG).

I just found a suitable region by trial and error (as per RickS's suggestion). I took a bunch of bias, dark, and light frames and looked at the image with stretching. You'd expect the Dark Dummy Pixels to look identical to a dark frame, brighter than Virtual Dummy Pixels, but less bright than light frame pixels... which looks like the vertical strip on the right edge of the frame.

You'd expect Virtual Dummy Pixels to look identical to bias frame pixels, but much dimmer than dark/light frame pixels - which is the horizontal strip below the rest of the image.

I only specified 60 extra overscan columns/rows in the driver (30 in this 2x2 binned image), but the first 6 rows below the light frame look like Dark Dummy Pixels (it's brighter than the rows at the very bottom)... which leaves 24 rows. I've probably chosen the wrong region, but it gives me much better dark scaling results so I'm happy for now

To find the right region, I'd recommend turning on overscan capture on the camera, and then record a few bias, dark, and light frames. Use screen stretching and the eyedropper tool to have a look at the overscan region to see which bits correlate with Virtual/Dark Dummy Pixels. You want there to be a reasonable amount of dark noise to differentiate between Virtual and Dark Dummy Pixels, so either take really long exposures (the light frame shown in the OP was 60 min), or run the camera warm - say +15 deg C - to exaggerate the difference.
Reply With Quote
  #10  
Old 01-10-2013, 03:25 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,581
Quote:
Originally Posted by naskies View Post
The overscan data download in the SBIG driver in no way correlated with the chip's pixel overscan geometry, i.e. I couldn't match specific overscan pixels to Dark/Virtual Dummy Pixels in the datasheet's diagram (neither could the person I emailed to at SBIG).
The datasheet for the KAF-16803 (Apogee U16M camera) similarly was little help when it came to interpreting the extra data appended to the frames by the Apogee driver. The driver remaps the data in some mysterious and undocumented way. Apogee support were vaguely helpful and after consulting with their engineering team confirmed what I had already figured out from inspecting the data but didn't tell me anything new. I think that overscan isn't widely used by non-professionals...
Reply With Quote
  #11  
Old 25-01-2014, 12:52 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 15,044
I am looking at using this feature now. I use a Starlight Express Trius 694
2750 x 2200 pixel and FLI Proline 16803 and Microline 8300.

I don't see an overscan item in the FLI Grab software which runs the camera for FLI.

The Trius uses a CCDsoft camera plug in and SX driver. I am not sure if there is something in the SX supplied software that may have an overscan feature.

Any suggestions on how to set this up?

Greg.
Reply With Quote
  #12  
Old 25-01-2014, 01:06 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,581
You could try asking on the SX Yahoo group or ping Terry Platt direct, Greg.

I do overscan calibration with my Apogee U16M and Maxim. The Apogee driver configuration panel has a check box called "Digitize overscan".

Cheers,
Rick.
Reply With Quote
  #13  
Old 25-01-2014, 01:09 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 15,044
Quote:
Originally Posted by RickS View Post
You could try asking on the SX Yahoo group or ping Terry Platt direct, Greg.

I do overscan calibration with my Apogee U16M and Maxim. The Apogee driver configuration panel has a check box called "Digitize overscan".

Cheers,
Rick.
Thanks Rick. I also posted this on the PI Forum as hopefully someone has been using this feature on a FLI camera.

I looked at the FLI Grab software and it does have a checkbox for entire sensor but not sure if that includes the overscan. I'll ask FLI.

Greg.
Reply With Quote
  #14  
Old 25-01-2014, 01:09 PM
troypiggo's Avatar
troypiggo (Troy)
Bust Duster

troypiggo is offline
 
Join Date: Nov 2008
Location: Brisbane, Australia
Posts: 4,790
Same with the QSI in Maxim. The feature has to be enabled in the driver/firmware of the camera.
Reply With Quote
  #15  
Old 25-01-2014, 02:06 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,581
Quote:
Originally Posted by gregbradley View Post
Thanks Rick. I also posted this on the PI Forum as hopefully someone has been using this feature on a FLI camera.

I looked at the FLI Grab software and it does have a checkbox for entire sensor but not sure if that includes the overscan. I'll ask FLI.

Greg.
I'm sure that FLI drivers provide access to the overscan, Greg. You need access to the overscan to do an accurate PTC analysis. Richard Crisp has an example using a FLI camera here: http://www.narrowbandimaging.com/ima...mments_web.pdf. Unfortunately, he doesn't show how you get access to the overscan.

Cheers,
Rick.
Reply With Quote
  #16  
Old 25-01-2014, 03:17 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 15,044
Quote:
Originally Posted by RickS View Post
I'm sure that FLI drivers provide access to the overscan, Greg. You need access to the overscan to do an accurate PTC analysis. Richard Crisp has an example using a FLI camera here: http://www.narrowbandimaging.com/ima...mments_web.pdf. Unfortunately, he doesn't show how you get access to the overscan.

Cheers,
Rick.
Yes Richard would undoubtedly know how to do it. I've posted the question.
The good thing is once its setup you shouldn't need to adjust anything in the future and it will be a permanent
feature of your processing.

Greg.
Reply With Quote
  #17  
Old 26-01-2014, 01:19 PM
White Rabbit's Avatar
White Rabbit
Space Cadet

White Rabbit is offline
 
Join Date: Feb 2007
Location: Sydney
Posts: 1,409
Hi Dave, or any one else that may be able answer.

I have at STF8300m as well and I notice that in your step by step guide that you say to enter 60 into the over scan region on the driver page yet your image of the driver settings has a value of 30. Then as I was reading down the thread you mention that you entered 30 for a 2x2 binning.
Does this mean that you need to change the overscan region every time you bin and un bin?

Thanks
Reply With Quote
  #18  
Old 26-01-2014, 03:08 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,581
Quote:
Originally Posted by White Rabbit View Post
Hi Dave, or any one else that may be able answer.

I have at STF8300m as well and I notice that in your step by step guide that you say to enter 60 into the over scan region on the driver page yet your image of the driver settings has a value of 30. Then as I was reading down the thread you mention that you entered 30 for a 2x2 binning.
Does this mean that you need to change the overscan region every time you bin and un bin?

Thanks
The overscan region is a fixed size so it will "shrink" proportionally when you bin. So, yes, you will need to modify the driver settings when you change binning if you want to use the whole overscan region.

You could leave the setting at 30 which would work binned x2 or unbinned. Whether you would lose anything by using only half the overscan region in unbinned mode I don't know. You'd need to do some data analysis to find out. It's probably worth doing some work to check the results anyway. I did quite a few integrations with and without overscan calibration before I was convinced it was always worth doing on my camera. The nice thing is that if you have the overscan data you can always ignore it and do conventional calibration.

Cheers,
Rick.
Reply With Quote
  #19  
Old 27-01-2014, 05:24 PM
gregbradley's Avatar
gregbradley
Registered User

gregbradley is offline
 
Join Date: Feb 2006
Location: Sydney
Posts: 15,044
I got a reply from Richard Crisp. He said that doing 20 darks would reduce noise more than averaging the overscan area as its only a narrow
sample.

I get from this there is a benefit from doing overscan callibration if your camera has a bias drift.

He recommended measuring it. So I will do a test and see how much the bias shifts over say 20 bias's at different times at the same temp.

Greg.
Reply With Quote
  #20  
Old 27-01-2014, 06:10 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,581
Quote:
Originally Posted by gregbradley View Post
I got a reply from Richard Crisp. He said that doing 20 darks would reduce noise more than averaging the overscan area as its only a narrow
sample.

I get from this there is a benefit from doing overscan callibration if your camera has a bias drift.

He recommended measuring it. So I will do a test and see how much the bias shifts over say 20 bias's at different times at the same temp.

Greg.
Hi Greg,

I have significant bias drift on both my KAF based cameras (SX H-18 and Apogee U16M) which is why I started digging into this in the first place. I agree YMMV depending on your camera/sensor and if your bias is stable then it is better not to do it.

In my case I found that overscan calibration made my master darks look much more linear (as did Dave) and the final integrations showed measurably better SNR.

From memory, the bias drift on the U16M was as much as 40 e- over a few nights. That's an order of magnitude more than the signal I've managed to tease out of a couple of targets. The jets of NGC 1097 are a good example. I don't think I could have picked up R4 without overscan calibration.

I'd be interested to hear how your cameras measure up.

Cheers,
Rick.
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 08:02 AM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Advertisement
OzScopes Authorised Dealer
Advertisement
Bintel
Advertisement
Lunatico Astronomical
Advertisement
SkyWatcher Australia
Advertisement
Meade Australia
Advertisement
Nitecore
Advertisement
NexDome Observatories
Advertisement
Astronomy and Electronics Centre
Advertisement