Overscan calibration to improve dark frame scaling (SBIG STF-8300M + PI)

[naskies]

Before I begin... thanks must go to RickS and the authors of various astro texts/webpages for informing me about overscan calibration and for helping me to get it to work in PI :)

Background: most of us are familiar with bias, dark, light, and flat frames. With bias frames, we can use scaled darks - i.e. take a single set of dark frames say 10 min in duration, and mathematically scale them for calibration against other sub durations (5, 10, 15, 20, 30 min etc).

The problem is that when you do the pixel math calculations by hand, you may find that it doesn't actually work. For example, a 20 min master dark frame might have only 150% of the signal in a 10 min master dark frame - rather than the 200% you'd expect according to theory.

It turns out there are two separate components to the bias signal: a fixed column-by-column (or row-by-row) pattern, PLUS an offset that affects every pixel equally but which changes with every frame. Master bias frames calibrate for the former, but for the latter we need overscan calibration.

Overscan calibration: The overscan region is a non-imaging part of the CCD sensor used purely for calibration. That is, these pixels don't receive light (they may be physically opaque up on the sensor), but they are still connected to the same circuitry for reading pixel data. Think of them as pixels that are permanently wired to take bias and dark frames (even when the rest of the CCD is actually taking a light frame).

It seems that until recently, most amateur astro imagers and camera manufacturers just ignored overscan pixels. Depending upon the camera manufacturer, they can be downloaded as extra rows and columns in an image with a bit of driver tweaking*.

The median value of the overscan pixels is then subtracted from all frames - bias, dark, flat, and light frames (so you need data reduction/processing software such as PixInSight that supports overscan to do this).

I've done some number crunching with dark frames on my SBIG STF-8300M operating at -10 deg C (see the table in the attached images). For simplicity's sake, just look at the "600 sec --> 3600 sec" row, which shows a mean ADU/pixel error of 199.9 and 32.8 for standard and overscan calibration, respectively. The 199.9 figure means that if I just use standard bias / dark / flat / light frame processing and scale a 10 min master dark up to 60 mins, the scaled dark will overshoot a 60 min master dark by 199.9 ADU/pixel on average. Using overscan calibration, the error is only 32.8 ADU/pixel - a 83.6% reduction!

Note: these numbers are based on the same image pixels - the only difference is whether the median overscan offset value is subtracted from each frame. Furthermore, overscan calibration would also improve the accuracy of flat frames.

Step-by-step: Here are the precise steps that I follow for an SBIG STF-8300M:

1. Upgrade to the latest camera firmware (mid-2013) - earlier versions of the firmware had bugs that resulted in grossly incorrect bias frames on my camera. Note, this will invalidate your bias/dark library.

2. Enable the overscan region in the SBIG 64 Bit Driver Checker Utility by clicking on the "Config Drvr" button and enter 60 into the Overscan Rows/Columns field. Turn off Auto Bias Level Correction*.

3. Launch MaximDL - it now detects the camera as 1706x1296 (instead of 1676x1266 using 2x2 binning). See the attached image - it adds dark bars to the right/bottom of the image. Capture bias + dark + light frames normally.

4. When reducing data in PixInSight, enable the Overscan function (see attached image) for all frames - bias, dark, flat and light. This crops the image back down to the original 1676x1266 pixels, and subtracts the median ADU from a 1676x24 pixel overscan region below the image. You may also need to set the Output pedestal to prevent negative values (I set it to 200 in this case).

5. You can integrate the resulting subs as per normal. I generated the numbers in the first table using the Statistics process in PI using the same subs, with and without overscan calibration.

Conclusion: The difference with overscan is not huge - you probably won't notice it in the final stacked output unless you're going really deep with the processing. That said, the KAF-8300M has a read noise a bit over 8e- (22 ADU) so the improvement here is significant. Since this reduces noise for "free" (takes no extra imaging time, equipment, or $$$), I think it's worth doing :thumbsup:

I'm more than happy to clarify anything that I've explained poorly :)


* I believe some manufacturers (QHY maybe?) have added overscan calibration functionality directly into the driver so that you don't have to do it yourself. I don't know any specific details.

** The Auto Bias Level Correction function - which I assume is meant to be automatically applying overscan calibration - on my SBIG STF-8300M has never worked correctly and gives erratic results, despite numerous firmware updates.

[gregbradley]

Great post, thanks Dave.

Greg.

[RickS]

Nicely explained, Dave!

I was poking around my narrowband Helix data today and noticed that some of the faintest nebulosity that I'm able to see clearly in the stacked images corresponds to only 25e- in a 30 minute exposure. When you compare that to the 40e- bias drift I have seen from the camera I think it makes a convincing case for overscan calibration, especially when working with dim targets.

Cheers,
Rick.

[naskies]

Thanks, gents. Yep, I agree for deep narrowband images it definitely seems worthwhile. For light pollution limited LRGB subs, it's probably not worth the trouble unless you're already doing it for NB.

I forgot to mention above - bottom line, with the 8300 chip (and presumably others too) I'd recommend taking dark frames at the longest duration you image at and scaling down for best results. For example, take a library of 30 min darks and scale them for LRGB subs at 5, 10, 15 mins etc. Doing it the other way (scaling up) result in much more error.

[RobF]

I salute you Dave!

[troypiggo]

Interesting. I'll need to go back and read this in more detail later.

Do you need to completely update your bias and dark libraries? Or does the clipping off of the overscan region take care of that?

[RickS]

G'day Troy,

You need to collect a new set of bias and dark frames with the overscan attached and then build new master calibration frames.

Cheers,
Rick.

[Bart]

In image 4 of the OP, on the line that says Source Region in the Overscan section of the Image Calibration tool, what is the 24 and how is this derived and what does it mean? Could you in general explain that section a bit more for me as my camera has an overscan option and am interested to give it a go.

Thanks.:)

[naskies]

There are several types of overscan pixels, but the ones I was interested in were (according to the KAF-8300M datasheet (http://www.truesenseimaging.com/all/download/file?fid=8.56)):

* Dark Dummy Pixels - working pixels that are never exposed to light; it's like normal pixels from a Dark frame but available in every image, and

* Virtual Dummy Pixels - simulated pixels that aren't connected to the vertical shift register (so bias + read noise is present, but no thermal noise or light data); which is like Bias frame pixels but available in every image.

The overscan data download in the SBIG driver in no way correlated with the chip's pixel overscan geometry, i.e. I couldn't match specific overscan pixels to Dark/Virtual Dummy Pixels in the datasheet's diagram (neither could the person I emailed to at SBIG).

I just found a suitable region by trial and error (as per RickS's suggestion). I took a bunch of bias, dark, and light frames and looked at the image with stretching. You'd expect the Dark Dummy Pixels to look identical to a dark frame, brighter than Virtual Dummy Pixels, but less bright than light frame pixels... which looks like the vertical strip on the right edge of the frame.

You'd expect Virtual Dummy Pixels to look identical to bias frame pixels, but much dimmer than dark/light frame pixels - which is the horizontal strip below the rest of the image.

I only specified 60 extra overscan columns/rows in the driver (30 in this 2x2 binned image), but the first 6 rows below the light frame look like Dark Dummy Pixels (it's brighter than the rows at the very bottom)... which leaves 24 rows. I've probably chosen the wrong region, but it gives me much better dark scaling results so I'm happy for now :)

To find the right region, I'd recommend turning on overscan capture on the camera, and then record a few bias, dark, and light frames. Use screen stretching and the eyedropper tool to have a look at the overscan region to see which bits correlate with Virtual/Dark Dummy Pixels. You want there to be a reasonable amount of dark noise to differentiate between Virtual and Dark Dummy Pixels, so either take really long exposures (the light frame shown in the OP was 60 min), or run the camera warm - say +15 deg C - to exaggerate the difference.

[RickS]

The datasheet for the KAF-16803 (Apogee U16M camera) similarly was little help when it came to interpreting the extra data appended to the frames by the Apogee driver. The driver remaps the data in some mysterious and undocumented way. Apogee support were vaguely helpful and after consulting with their engineering team confirmed what I had already figured out from inspecting the data but didn't tell me anything new. I think that overscan isn't widely used by non-professionals...

[gregbradley]

I am looking at using this feature now. I use a Starlight Express Trius 694
2750 x 2200 pixel and FLI Proline 16803 and Microline 8300.

I don't see an overscan item in the FLI Grab software which runs the camera for FLI.

The Trius uses a CCDsoft camera plug in and SX driver. I am not sure if there is something in the SX supplied software that may have an overscan feature.

Any suggestions on how to set this up?

Greg.

[RickS]

You could try asking on the SX Yahoo group or ping Terry Platt direct, Greg.

I do overscan calibration with my Apogee U16M and Maxim. The Apogee driver configuration panel has a check box called "Digitize overscan".

Cheers,
Rick.

[gregbradley]

Thanks Rick. I also posted this on the PI Forum as hopefully someone has been using this feature on a FLI camera.

I looked at the FLI Grab software and it does have a checkbox for entire sensor but not sure if that includes the overscan. I'll ask FLI.

Greg.

[troypiggo]

Same with the QSI in Maxim. The feature has to be enabled in the driver/firmware of the camera.

[RickS]

I'm sure that FLI drivers provide access to the overscan, Greg. You need access to the overscan to do an accurate PTC analysis. Richard Crisp has an example using a FLI camera here: http://www.narrowbandimaging.com/images/ptc_talk_wsp_2009_crisp_final_comme nts_web.pdf. Unfortunately, he doesn't show how you get access to the overscan.

Cheers,
Rick.

[gregbradley]

Yes Richard would undoubtedly know how to do it. I've posted the question.
The good thing is once its setup you shouldn't need to adjust anything in the future and it will be a permanent
feature of your processing.

Greg.

[White Rabbit]

Hi Dave, or any one else that may be able answer.

I have at STF8300m as well and I notice that in your step by step guide that you say to enter 60 into the over scan region on the driver page yet your image of the driver settings has a value of 30. Then as I was reading down the thread you mention that you entered 30 for a 2x2 binning.
Does this mean that you need to change the overscan region every time you bin and un bin?

Thanks

[RickS]

The overscan region is a fixed size so it will "shrink" proportionally when you bin. So, yes, you will need to modify the driver settings when you change binning if you want to use the whole overscan region.

You could leave the setting at 30 which would work binned x2 or unbinned. Whether you would lose anything by using only half the overscan region in unbinned mode I don't know. You'd need to do some data analysis to find out. It's probably worth doing some work to check the results anyway. I did quite a few integrations with and without overscan calibration before I was convinced it was always worth doing on my camera. The nice thing is that if you have the overscan data you can always ignore it and do conventional calibration.

Cheers,
Rick.

[gregbradley]

I got a reply from Richard Crisp. He said that doing 20 darks would reduce noise more than averaging the overscan area as its only a narrow
sample.

I get from this there is a benefit from doing overscan callibration if your camera has a bias drift.

He recommended measuring it. So I will do a test and see how much the bias shifts over say 20 bias's at different times at the same temp.

Greg.

[RickS]

Hi Greg,

I have significant bias drift on both my KAF based cameras (SX H-18 and Apogee U16M) which is why I started digging into this in the first place. I agree YMMV depending on your camera/sensor and if your bias is stable then it is better not to do it.

In my case I found that overscan calibration made my master darks look much more linear (as did Dave) and the final integrations showed measurably better SNR.

From memory, the bias drift on the U16M was as much as 40 e- over a few nights. That's an order of magnitude more than the signal I've managed to tease out of a couple of targets. The jets of NGC 1097 are a good example. I don't think I could have picked up R4 without overscan calibration.

I'd be interested to hear how your cameras measure up.

Cheers,
Rick.

[gregbradley]

I guess the only way I'll find out is to measure it to know.

When you say bias drift how did you measure it? Did you wait for 20mins for the camera to settle down after it reached its cooling temp?

Did it vary with the ambient temp falling as the evening progressed?

Richard also says bias will be different at the start and at the end of a read out process. So there is a variable there as well. I don't know how you measure that if you can at all.

Greg.

[RickS]

I measured the average bias in a selection of frames using the Statistics process in PixInsight. I spent several days collecting calibration frames interleaving bias frames and darks of various durations. I was indoors in a room that doesn't vary in temperature a lot (it is embedded in the side of a hill) and didn't start the calibration run until well after the camera temp had stabilized.

I didn't notice any significant variation in bias from the start to the end of the frames but I'll go back and have a closer look. In PixInsight it is easy to set up previews over different parts of the frame and grab stats for each preview.

Cheers,
Rick.

[gregbradley]

I think he was referring to the readout process from the reading at the beginning of a single image to the end of that single image.

Also to falling temps as the evening progresses.

I'll do an experiment to find out next time I am using those cameras.

Greg.

[RickS]

Yes, that's what I thought he meant. He's expecting to see a variation in average bias between the first rows and the last rows due to the time delta between when they are read out, presumably due to accumulating a little extra dark current. I can check that by measuring different sections of the bias frames.



He's suggesting that some of the bias drift may be due to thermal effects in the external support circuitry which is not kept at a constant temp like the sensor. I agree that's quite plausible but I haven't tried to test it. It would be interesting to know what causes the bias drift but my first priority was figuring out how to compensate for it.

Cheers,
Rick.

[RickS]

Greg,

I checked my bias master. There is a slight difference in average value from the first rows to the last rows but it is tiny - about 0.015 ADU compared to a bias value of 1220 ADU. This is equivalent to about 1.5 seconds of dark current at -30C. That could be about how long it takes to download the data from the sensor into the RAM buffer on the camera.

Cheers,
Rick.

[Shiraz]

very informative thread - thanks guys.

Presumably bias drift will vary with camera design. For interest, I looked up the recommended support circuitry for a couple of popular CCD chips - one manufacturer suggested what appears to be a thermally compensated output buffer, the other a straight emitter follower, with no compensation. One would likely drift only a little bit, the other probably more so.

It might be informative to have a more complete list of cameras that could benefit from overscan compensation. Will do a few tests on the H694 bias stability and post here - it doesn't produce overscan data as is, but I am not sure how much it drifts.

As I understand it, once you have reliable set of darks (for which you might need overscan compensation), the issue is dealt with - all you need overscan for is to get the right dark current to scale and subtract. Bias drift in the lights will only show up as a variable offset, which can be easily dealt with - is this a reasonable statement? EDIT: of course this only applies to pretty pictures, not science data - overscan compensation would be essential for some forms of photometry.

regards Ray

[RickS]

Ray,

I do overscan calibration of all my data - calibration frames and light frames. PixInsight subtracts the median of the overscan region from all of the data pixels and then everything else happens as usual. This is intended to remove any bias drift from the data before it is processed.

Cheers,
Rick.

[gregbradley]

How do you measure bias drift? Do you check background ADU of a series of bias images taken at the same time and same temp?

I get the idea from Richard that it may also tend to shift when ambient temps change.

Here is a copy of a post by Richard today where he measured his Microline. Bias seems extremely stable.
So this overscan callibration may only be of benefit if your camera has bias drift.



Richard Crisp
Today at 3:14 AM
View Source


0 Attachment











I did a little bit of temperature testing in an older ML4022 to measure the bias value under different ambient temperature conditions and different sensor temperatures


I operated the ML4022 in three different ambient temperature conditions: Room, Refrigerator, Freezer


I didn't have a quality thermometer handy but would estimate the ambient temperature was about 70F in the room. I guess the Refrigerator temp was around 38F and the Freezer was probably below 20F maybe as cold as zero F.


here are the results for -20, -25, -30 and -35C sensor operating temperatures


Bias Values (ADU)
op temp-20-25-30-35freezer977977977979refrigerator98 7988989991room991991992993

Since my night temps usually fall within the range of 70-40F the room temp and refrigerator temps cover my normal range. It is rare for me to encounter really cold freezing temperatures.


looks like the spread for my expected conditions is in the range of 2-4 ADU.

Richard Crisp




Greg.

[RickS]

I measure drift by looking at the mean value of each bias frame (or a smaller section of each bias frame if you prefer - Dave and I did some tests on a 100x100 pixel area at the centre of each frame). The time and ambient temp (did you mean ambient or sensor temp?) shouldn't matter. If your bias is not stable regardless of these then overscan calibration is likely to be of benefit.

Ambient temperature is definitely a possible cause of bias drift by affecting the electronics external to the temperature regulated sensor. I didn't ever see an obvious correlation in my results but that's not conclusive.

Both Dave and I did tests that demonstrated a significant benefit to overscan calibration with our cameras - a SBIG STF-8300M in his case and an Apogee U16M in mine. With overscan calibration we got a much more linear looking growth of dark current over time and could also measure a reduction in effective read noise (bias drift can't be distinguished from read noise). YMMV of course, and as I said before: if you don't have bias drift then there's no value in overscan calibration.

I'm pretty busy with work stuff at present but once I dig my way out from under that I'd be happy to post some numbers and graphs. Dave might weigh in at some stage too but he's in the middle of a significant lifestyle change and may be too busy to spend time on IIS for a while.

Cheers,
Rick.

[White Rabbit]

Quick question re meridian flip when using the over scan feature.

I'm guessing that when it comes to stacking images that have overscan visible I need to stack each side of the meridian separately or the overscan regions will not align? This also brings up another point what if I dither then the overscan regions wont line up either? I'm missing something arnt i?? I need to go back an re read the entire thread.. don't I. Ok I'll go back a re read the entire thread.

...

[RickS]

The overscan is used during calibration of individual subs then cropped off before they are registered and stacked. It doesn't have any impact on meridian flips or dithering.

[White Rabbit]

Ann, perfect thanks. One more piece to the puzzle added.

[pvelez]

Dave

this is extremely helpful - I've been playing with calibration with PI last weekend as I was encountering readout issues.

I have a SBIG STX16803 - is the 60 pixel overscan setting appropriate for that camera?

Pete

[RickS]

Hi Pete,

Dave has been pretty busy lately so I suspect you won't hear back from him soon. I also suspect he won't know the answer to your question. You should ask SBIG and/or poke around the extra data in the frame to figure out where the overscan data lies. On my 16803 camera (Apogee U16M) there is a set of 10 columns at the far right of the frame that do the trick but the SBIG driver will almost certainly map things differently.

Cheers,
Rick.

[troypiggo]

My QSI583 (Kodak KAF 8300 sensor) has 279 columns on the right. Spoke to QSI's Kevin Nelson about it, and he recommended just using the right-most 100-150 columns. So it varies from sensor to sensor, and possibly(?) camera manufacturer to manufacturer.

[pvelez]

Thanks Rick

I think it was the discussion about real vs dummy overscan that threw me. I can access my obs PC from work so I'm playing with this as a I type.

Looking at a 1200 second dark, I have a dark border to the right and bottom of the frame. However at the right side, there is a brighter line (several pixels wide) between what appears to be the edge of the image and the dark border. I wonder if this is the readout channel for the chip. I've posted a pic.

Interestingly, I have no dark border at the base of a bias frame, only to the right. Also attach a pic.

How do I distinguish the real overscan from the dummy overscan?




Sadly, SBIG are less forthcoming - though there may be something deep in the bowels of the SBIG site - off to do some digging now

Pete

[pvelez]

oops - that is the bias - here is the dark

Pete

[pvelez]

Did some more digging - the KAF16803 fact sheet suggest that its only a 20 pixel border to the left and above/below the chip that are useful for overscan and 9 pixels to the right.

Here is the link

http://www.ccd.com/pdf/ccd_16m.pdf

My head is spinning - so I guess its time to use the time-honoured trial and error method

Pete