Trius + CCDStack: Is This Correct?

[PRejto]

I would appreciate some feedback on this work flow in the beginning steps of calibration using subs from my Trius. I don't want to subtract dark frames. However, I do want to get rid of hot pix and use a flat frame master using CCDStack.

1. Select the master bias frame as a "dark" in CCDstack calibration tool
2. Select "none" for adaptive subtraction.
3. Select proper master flat.
4. run calibration
5. Go to Process/Data Reject/Hot Pix strength 4, run
6. Interpolate rejected pix: width .2, iterations 3
7. Proceed to registration.

Or, is it better to make a bad pixel map first from a master dark and as a first step use the pixel map to eliminate the hot pix by interpolation followed by steps 1-4?

I'm also a bit confused about the bias subtraction. When making the master flat CCDStack asks if I want to subtract the bias. I've always indicated "yes." I assume that by subtracting the bias during calibration (instead of a dark) this does not present a conflict.

Any advice much appreciated.

Thanks,

Peter

[PRejto]

I have done a bit of testing and I'm unsure about these results. The raw sub is a 12 min OIII. I took 50 bias frames, 20 x 12 min darks, and 30 flats. Perhaps with more frames I wouldn't see such a big hit to the s/n.

The s/n seems to really suffer if I subtract either a dark or a bias. If I apply just the flat without a dark or bias the flat does not work properly though the s/n is only reduced a little. I tried making a flat without subtracting the bias and it also doesn't work properly. Here are the s/n. (Are these values in line with what you see with your Trius type camera?)

Raw Frame s/n = 2.61
Flat calibratio only 2.45
Bias only .43
Dark only .38
Bias + Dark .38
Dark + Bias + Flat .39
Bias + Flat .43

Are the benefits of applying bias or dark worth the price of the s/n reduction?

I must apply a flat. However I cannot apply the flat master without a bias or dark subtraction. Is there a reason why this isn't working? My flats were made at ca 35,000 adu

There is one other odd (?) thing I don't understand. In the info window that also measures s/n the raw image shows a range of ADU values min= 1330, max = 64999. After applying a bias: min= 76, max = 63733. After the dark frame: min= -14,664, max 63,705. I'm not sure what the negative number means....

Thanks for any help!

Peter

Edit: I started this thread thinking it was going to be a software question about using CCDStack. It has clearly turned into a more general processing question. Should I move the thread?

[jase]

Hi Peter,
Your steps 1 to 4 will work fine. For the Trius with Sony sensor just subtracting the bias is a good option. I'd be inclined to build a bad pixel map but the process for doing so in CCDStack is a little convoluted. I ended up have to script mine to get the best result. Other apps such as MaximDL, PI, etc are more intuitive with bad pixel maps.

With regards to flat subtraction. Only need to subtract the bias assuming your flat sub exposures are relatively short. Say under 30 seconds or so. Being so short, a bias master is a better match. If you've got some very long flat subs, you may want to consider dark subtraction.

Just remember to not subtract the bias from the flat twice per the attached.

[jase]

35k ADU sounds quite high for a master flat. What is the well depth of the SX Trius? Have you done a photon transfer curve (PTC) to measure the full well saturation. You'll often find is different from the manufacturer specs. If you read the Onsemi/Trusense specifications the ccd testing is done at 27 and 40 degrees so not representative of the effects of cooling. I understand Sony don't publish any specs hence the reason a PTC is valuable.

The negative value may be due to the fact that the pedestal value is not being considered after the operation. Under adaptive subtraction, have you tried to use pedestal or average bias = 100?

[PRejto]

Hi Jase,

Thanks so much for your suggestions!

I will check out the well depth and also retake the flats at a lower ADU. But, in saying that, if I load a Trius photo into CCDStack over saturated stars register 64,000+ADU. And, dimmer stars are registering a correspondingly lower value. So (and I may not be understanding something) an average ADU of 35,000 in a flat ought not to be saturated.

I didn't subtract the bias 2X in the calibration.....

I'm unsure what it means to use an average bias of 100. I would think it less accurate than using an actual bias master. Perhaps I need to make adjustments in the camera Manager of CCDStack. Anyway, I'll play around with both.

Thanks again,

Peter

[jase]

Hi Peter,
Yes, my suggestion is to lower your target ADU for flats. If we take the stats from the SX site. Well depth 17k-e with a gain/ADU of .3 then we can quickly derive a target ADU for flats.
(full well/gain)*Z%=target flat ADU
(17,000/0.3)*35%=19,833 ADU
Z is the well capacity you are trying to reach which I typically work on a value between 30% to 50%. 50% gets you close to 28,000 ADU. The above is quick and dirty way given the stats off the SX site may not apply to your camera. The figures are generalised for the model. The camera gain is likely to be correct (.25 rounded to .3) but your well depth could be greater or smaller than whats listed. Testing your camera's capabilities through a proton transfer curve is the only way to get accurate stats unique to your camera. It should also be highlighted that ABG camera go non-linear as they being to reach saturation point. i.e. they being bleeding charge. Many will begin bleeding before they reach the well depth which results in the camera going from linear to non-linear. You want to make sure your target ADU for flats is well away the non-linear stage. I doubt you would be hitting it at 35k ADU anyway but I probably wouldn't go higher.

The above said and done, I don't think flats is your main challenge. The flats could well be working as expected without changing anything. Its the desire to not inject noise into what is an already very quiet ccd. If you are hitting the right flat target ADU and have plenty of subs, noise will not be injected. In fact your stats indicate this;
Raw Frame s/n = 2.61
Flat calibratio only 2.45
Bias only .43
Dark only .38
Bias + Dark .38
Dark + Bias + Flat .39
Bias + Flat .43
The last test indicates that no further noise is being injected by the flat. Please make sure you are measuring the exact same X and Y coordinates in each image to ensure you are obtaining comparable readings. Did the flat successfully apply i.e. can you see the difference and confirm in the fits header that it was applied. What is missing from the tests is Dark + Flat.

With regards to the negative number, was this for the same area you were previously measuring or the image as a whole? If its negative, use pixel math to make it a positive value otherwise you'll have data rejection troubles. The subtraction of a dark shouldn't do this however. Strange behavior. Note that CCDStack does not add a pedestal to the image data by default with exception to 16bit colour FITS and TIFF which are given 100 ADU pedestal. Until you nut out whether the noise from the bias can be reduced, perhaps stay away from the adaptive subtraction pedestal value.

I think I'll need to check how CCDstack measures the S/N in the information window. From memory its mean divided by standard deviation of non rejected pixels within the selected area. A basic approach but is still comparable.

[PRejto]

Hi Jase,

I'm going to redo my flats at a lower ADU. Also, I noticed that the camera manager doesn't have correct values at all for the Trius. I'm going to calculate the exact values for my camera and just start over. I suspect I may have made some sort of error re the bias frames too (which of course would totally mess up the flats and that might explain why the current flats are not working). Does this bias master look reasonable?

Thanks again,

Peter

[jase]

Hi Peter,
The gain, read noise and pedestal values under CCDstack camera manager are really only beneficial for data rejection algorithms, specifically sigma-rejection and poisson-rejection. The default values will work ok but if you are keen, you'll likely want to reflect the correct values.

I wouldn't get too hung up on the S/N stat as you may find yourself chasing your tail when in reality signal is going to be gained and noise reduced through stacking (square root).

The bias master looks fine but I don't have anything to compare it against. The dark frame test no doubt already included the bias. It appears to give the lowest S/N.

[PRejto]

Hi Jase,

Thanks for the additional feedback.

I redid the bias frames and nothing is different. I'll do the flats next.

Just one more question (sorry!). If I create a map of hot pix I would do that from a dark frame I assume. Would the order of processing be to first use the map and then calibrate with a bias (acting as a dark) + flat field? I assume one could not use a dark as that would try to remove the pix twice and leave black marks on the subs.

Thanks,

Peter

[jase]

Hi Peter,

Yes, you're correct. You can use either a master dark or a light frame that exhibits the bad pixels/columns you wish to reject. The master dark will have the bias included so you would think that would be a bad thing i.e as you point out, you would be be subtracting defects twice. Technically though a bad pixel map isn't subtracting anything, it just flags the pixel/column defects as a 'missing value' i.e. highlights them in red indicating they are ready for interpolation. CCDstack has info on building the map using the data rejection process. I would build the map with a master dark first to see how that works for you.

In your case, it would go something like this.
1 Calibration (bias and flat)
2 Reject bad pixel map (interpolate rejected pixels)
3 Image registration (method of choice, large stacks go with nearest neighbour, small bi-cubic)
4 Normalisation (manually select background and highlights)
5 Data rejection of choice. (STD large stacks, Poisson small stacks)
6 Mean combine (or sum as if need be for working with faint narrowband signal)

Any further outliers in your data left over from steps 1 and 2 will get rejected in step 5 - assuming you are dithering. I don't know many that don't dither between subs these days.

[PRejto]

Jase,

Thanks for that plan. I can see why one should calibrate before using the pix map!

I made a new bias master with 200 subs. When I use it as a Dark + new flat frames (at ca 19,000 adu) I do get a good result but take a big hit in s/n. The new FF will not work alone and actually makes the dust crud even more visible. I must use a bias or dark, or both to get a usable result.

Interestingly using the bias + Flat I do not get a negative number for the min value. But if I use the dark frame the min =-13000. Also, CCDStack scales the dark to 95% even though it is a 12 min dark and a 12 min light both at -15 cent. I would have though the bias would not be used in that situation even though it was loaded in the calibration tool.

I ran the numbers on my camera and got Gain: .322221447, Read Noise: 5.0148677799. I tried to read about how to measure/perform the PTC. I immediately felt dizzy (and stupid).

Peter

[gregbradley]

I refined my callibration steps using my CDK17 which is sensitive to good flat fielding. A slightly off master dark, or a weak flat or off temperatures or no bias subtract to the flat master at callibration would wreak havoc.

I do an average combine of flats - no bias subtract, to form a flat master.
Ray did an article about the number of flats to use for an ideal flat on this site a few months back.

I make a master bias and have that subtracted when applying the flats. This seemed to give a better result.

Doing dark subtraction seems to make images noisier with the Trius. The bias subtract though gets rid of a white line along the left side of the image. This does not seem to show in final images if not removed but probably better if it is. So I have also started doing a bias subtract as well as a flat if the image has dust donuts. If no dust donuts (better to clean filters carefully with a photographic cleaning cloth and a blower) then no flats may give the better image as it introduces some noise.

There is a paper by Richard Crisp about flat ADU levels. He advises the opposite here of doing higher ADU values like 30-40K. I am sure its on his website if you look for it. I tend to go for between 20-30K but in the end I check to see what works and adjust. Sometimes I went for lower ADUs with the CDK17 like 20K as 30K sometimes overcorrected and made the corners too bright.

Hot/cold pixel removal during data rejection seems to clean up anything remaining. I used to see the odd coloured dots in final images before I used that now I rarely do.

As the subs tend to be shorter with this camera lots of subs tends to make these statistical methods work better as they all work by identifying what is an outlier (outside the "norm") one way or the other.

I find median combine works best for combining images. I believe bi cubic sampling is a bit old hat and not best (I could be wrong). Lanzcos 36 is better but its slow. How much difference these little points make is debatable to the final image quality but I suppose every little bit helps.

I find my resulting master Trius images are nice and clean this way.

If I do use flats I may be inclined to use a master dark. I find though these tiny Sony chips tend not vignette to any real degree with the scopes I have been using.

Sony chipped cameras require a different approach than Kodak based cameras which are way noisier in comparison with usually 3 to 4 times the read noise and 2/3rds or less the QE with way weaker QE in Ha and O111, S11. If Sony could only make a full frame ExHad 11 CCD nobody would use the Kodaks.

A defect map would be an interesting alternative to a dark subtract. If you do make one I'd like to hear of your results. I find though the very few hot pixels are gone by the time I do the above.

Greg.

[PRejto]

Hi Greg,

I appreciate your feedback! Especially since you are using the same camera.

What I cannot understand, and perhaps I'm not grasping something basic, is why I'm unable to calibrate using only a master flat (I have 2 different masters one at 35K adu, the other at 20K adu). That is, without either a dark subtraction, or a bias subtraction (acting as a substitute dark) in the calibration tool. If I use a "dark" I get a good result with either flat. If I disable use dark in calibration I get a terrible result. I've tried both ways. 1. subtract bias when making the flat, 2. by subtracting the bias when running the calibration tool.

Should it be possible to calibrate using only a flat and get a good result?

I easily made a pix map from a dark frame. I've tried applying the map on the dark frame (as a test) and to raw and calibrated subs (calibrated by using the bias as a dark). The remove pix seems to work great. (I selected interpolate.)

Ideally I'd love to be able to just apply a flat (without using a dark or bias frame) and the pix map. I think it would give the best results in terms of s/n but so far this seems not possible.

Peter

[gregbradley]

I am pretty sure you can simply apply a flat only. I'll try that once I unpack from my trip and have that computer out. With my CDK flats only often do not get the right result to fully correct vignetting and hot centre areas. A correctly matched temperature and exposure length master dark that is high quality and fairly fresh (not 1 or 2 years old) with a good bias worked really well.Perhaps it depends on your scope but TEC APO's are piece of cake to calibrate as are most refractors.

You can select your master bias for the dark in the dark dialogue box. They are both a subtraction so no difference to the image.

CCDstack has gotten more options over the years, which complication often fights you as with other programs. The main change seems to allow auto selection of darks, biases and flats which I normally select manually so I guess I must be in the minority of users there.

I am not sure how the pixel map is applied. That must be a separate menu box? I wonder if you should apply it first or second. Probably first. Flats are merely correcting the vignetting and dust donuts. Most CCDs seem pretty evenly illuminated. Sometimes a corrector gives an uneven illumination with a definite brighter centre like a reverse donut. Tak BRC250, CDKs, perhaps some reducers do this.

Greg.

[jase]

Regardless of what camera you have, every pixel in a CCD exposure starts out at its bias level. You will always need to subtract the bias whether done using the bias frame or the bias as contained in a dark frame.

When you bias subtract your flats, this only applies to the flat subs to produce the flat master. You can't use the flat master to subtract the bias in light frames, so you must either use a bias or dark to remove the bias from the light. This is the reason why when you only subtract the flat, your light frames are still a mess. You've not removed the bias.



Yes, but in order to determine the right value you need a PTC for your camera. As Richard's paper mentions, you don't want to be taking flats in the ADU range where the camera goes non-linear. Its just before that to maximise signal and suppress noise.

[jase]

I think you may well be in the minority there Greg. The Flat, Bias and Dark manager is extremely versatile in CCDStack. I batch process calibration similar to PI. You don't need to only load up specific filter data for calibration. You can load all the subs regardless of filter and calibrate them. CCDStack will work out which bias, dark and flat to apply. Flats can be matched on filter and position angle if you've got a rotator. Very easy stuff. Adam Block has an addendum for CCStack version 2+ that goes into detail on the power of the CCDstack's calibration - https://vimeo.com/40758444
password skycenter

[RickS]

Median combine naturally rejects outlying pixel values but produces lower SNR than an average combine (about 20% less.) Average combine with careful rejection can give you the best of both worlds.



A flat field PTC will also allow you to figure out how many flats are enough. In the case of my KAF-16803 the sweet spot is around 500,000e- which is around 12 flats at half full well.

Cheers,
Rick.

[PRejto]

Hi Greg,

I'd be quite curious about your result of only calibrating with a flat.

I think the pix map ought to be used only after applying a flat + bias (as substitute dark). Obviously you wouldn't use the map before or after using a dark as that would attack the same pix 2X. I would apply it after the flat since the flat will remove crud and light irregularities. My reasoning would be that more correct pixels would be neighboring the removed pix for the interpolation. There are probably better reasons but that's the one that comes into my head.

I've been redoing my darks this morning and I'm noticing something that seems odd. I've done 22 12 min darks. After the first 10 or so the mean value starts progressively dropping, as does the min value (but not the max). I don't see why this should be happening. It's as if the longer the CCD is running the lower the min count is: Started min = 1232, after 10 frames min= 1225, after 20 frames min = 1209, after 27 frames min = 1189. This seems really odd! Any idea?

Thanks,

Peter

EDIT: I wrote to Terry Platt at SX. He said it is "just the reference voltage in the A-D that is drifting slowly as the chip warms up. It doesn't affect the darks – it just shifts the bias point slightly."

Hmmm.I'm not sure why it wouldn't affect the darks. Wouldn't the average total ADU be drifting lower and thus effect the amount that the dark applies in calibration?

[PRejto]

I learned something today from the CCDWare Forum.

The negative value I am seeing after dark subtraction can be cause by a pix on the dark that has a higher value than seen on the light frame. After the subtraction there is a negative value. To remove the pix(s) you do a reject with a range >-(whatever the negative value is) <0, then interpolate. After doing exactly that I got this result:

Image File Trius_M16_Filter_6_.00000007
rectangle {X=9,Y=0,Width=2741,Height=2200}
diagonal 3,514.70
# pixels 6,030,194
# rejected 6 (0.00%)
mean 107.99
STD 365.66
S/N 0.30
median 97.22
int mode 92
min 0.01
max 63,132.63

So, I had 6 pix giving this problem.

[gregbradley]

Yes, but in order to determine the right value you need a PTC for your camera. As Richard's paper mentions, you don't want to be taking flats in the ADU range where the camera goes non-linear. Its just before that to maximise signal and suppress noise.[/QUOTE]

His article also points out that most are taking their flats at too low an ADU and that most CCDs are very linear up to near the top of their range.

I suppose its trial and error to find an exact right flat values for your setup and scope. I'll have to do a PTC as I haven't done one before and he often refers to them.

You can also overcorrect with flats so the corners become too bright. So its a bit of an art rather than a precise science in my opinion.




Hehe, yes I was being lazy there. Thanks for the link. Perhaps I'll be selling how great it is next week!





I just tried it and you can do a flat by itself. In my case flats don't do much as there is virtually zero vignetting as long as my filter are clean.

Yes that is bias drift and not a good thing. Rick was using a bias drift bias drift super bias that uses the overscan region of the sensor. I am not sure how you access the overscan region to do that. I am sure there is a tutorial somewhere on the PI site.




How do you find out you have negative values?

Greg.

[RickS]

The method for enabling overscan depends on the camera driver. In the case of my Apogee there is a "Digitize Overscan" check box in the Apogee GUI under Maxim.

The only software packages that I know support overscan calibration are PixInsight and Mira.

It's certainly worth considering on any camera that shows significant bias drift. Naskies had an SBIG camera with a KAF-8300 that also produced better results using it, so it's not just me :)

Cheers,
Rick.

[PRejto]

I just open the dark in CCDStack., select all of the image, and the info box pops up. It's right there as a min value.

After the comment of Stan that a bias (as a dark) or a dark is required to properly subtract a flat I'm wondering how you are successful. Perhaps yours looks OK because you have a lot of signal. The image I've been unsuccessful with is an OIII with low signal. As I've said it flats fine with a dark....

Have you tried to make a pix map yet? It's pretty cool to see it work. I used a dark frame and did a hot pix rejection. It grabbed all the pix and turned them red. They really stand out against the dark. That gives you the map. Just for fun I used the dark to try interpolation. Like magic they were history and my dark looked a lot like a bias frame..

Peter

[gregbradley]

[QUOTE=PRejto;1192585]I just open the dark in CCDStack., select all of the image, and the info box pops up. It's right there as a min value.

After the comment of Stan that a bias (as a dark) or a dark is required to properly subtract a flat I'm wondering how you are successful. Perhaps yours looks OK because you have a lot of signal. The image I've been unsuccessful with is an OIII with low signal. As I've said it flats fine with a dark....



HI Peter,

I do use a bias when flat fielding. Its in the main calibration box as a checkbox to subtract a bias. The difference was to not subtract the bias when making the master flat but rather subtracting a master bias from a master flat at the time the flat is applied. It seemed a fine distinction but it gave better results on the CDK which as I say is fussy compared to refractors which are not fussy.


By the way flats are divided not subtracted.
That's why a flat with a bias does not do the bias on the image twice when you also do a bias subtract.

Thanks for the info on the hot pixel map I will try that. Does it also do a dark column?

I think also we should make it clear this discussion is primarily about calibrating a Sony sensor not a Kodak one. Kodaks need darks, biases, flats etc. A clean filtered Sony at around -20C may only need a bias and perhaps a pixel map.



Thanks Rick.


A question.

If you rotate your camera and filter wheel, reducer etc together but not the telescope are the flats still valid? Adam Block and CCDstack imply they are not but I got the idea from a Roland Christen post they are. I guess the camera and filter wheel for dust donuts are still in the same orientation. My experience here would be what Roland says that the uneven illumination is more from the camera, filters dust and anything in the optical path like an OAG prism. If they al rotate together then the scope is the same as any scope I have used seems very evenly illuminated except some have a bit of bright centre spot but that rotates evenly.

Greg.

[RickS]

Greg,

My experience is that rotation of the whole camera/FW/OAG assembly doesn't affect flats on the systems I use. However, this would not be the case on a scope where the rest of the optical system introduced some vignetting, especially if the camera isn't perfectly centred. I guess it's something you need to test. The old calibrate a flat with a flat trick would come in handy...

Cheers,
Rick.

[PRejto]

Hi Greg,

This is what Stan wrote:

"Sony CCDs have remarkably uniform dark current (pixel-to-pixel) and often do not need dark subtraction. In fact, dark subtraction may inject more noise than it cures (that's why it is advisable to use a very high quality dark). But when avoiding dark subtraction, it is necessary to remove the bias or bias level (pedestal) in order for flat fielding to work properly. And it might be useful to mask out a few consistently bad pixels. This can be implemented in CCDStack."

So, I don't think he is talking about the choice of bias subtraction whilst making a dark vs. doing the bias subtraction when running the calibration tool. I think he is speaking about actually using a dark (or a bias frame as a dark).

Yes, you can make a pixel map and map the columns. You just use different rejection tools. There are examples of doing it free hand in Adam Block's tutorial.

Peter

[gregbradley]

Thanks Peter. That's good data. Is this on the support forum for CCDWare?



Testing is good to know for sure. Not sure what the calibrate a flat with a flat test is. They null each other out?

Greg.

[PRejto]

Hi Greg,

Yes it is on the CDDWare forum.

I'm also curious what a flat with a flat test is.

Peter

[RickS]

The purpose of flat calibration is to undo the effects of imperfect optics (vignetting, dust motes, etc.) and also variations in the sensitivity of the sensor. So, if you calibrate a flat with a flat or flat master you should get a result which represents the original even illumination which created the flat. If there are significant variations in illumination or visible structure then there is a mismatch.

So, take some flats at one orientation, rotate camera/FW/OAG and take some more. If a master flat from the first orientation produces a clean result when calibrating a flat from the second orientation then you know that your flats aren't affected by rotation (maybe try a few different orientations just to be sure...)

Cheers,
Rick.

[PRejto]

Thanks Rick. That makes a lot of sense and worth trying!

Peter