Darks for your Flats!!!

[avandonk]

I have found that taking darks just after your flats really helps. You should correct your flats with the darks in the usual way.

Think about it the dark for your light removes the bias as well. The flat still has the underlying bias DSLR banding pattern. You are then modulating all your light data with this artefact.

Try it!

It is really most noticeable with really faint data and extreme stretching.

Bert

Below is a crop from the corner of a one second exposure (dark) at 100 ISO and 22C stretched to show detail.

[[1ponders]]

spot on Bert, even if the flats are only short exposures. Longer exposures >1 sec should be done anyway for noise removal of the flats otherwise the noise from the flats will be divided into the lights. Removing the bias from the darks is crucial if you want them to perform correctly.

[dugnsuz]

I take my flats using the "P" setting on the 40D - camera decides best exposure time etc for itself. If I put the lens and viewfinder caps on after taking my flats without changing settings, the camera will take darks of different exposure times from the flats.
Will this matter, or do the darks have to be a minimum time to work?
Cheers
Doug

[[1ponders]]

Darks must be the same exposure time as your flats Doug. You'll need to identify your flat exposures and apply that to your darks as well.

[Octane]

Doug,

Think of your flats as lights. Your darks should be exposed equally.

I've found, with 350Ds, anyway, that taking bias/offset frames has added noise to my images. I don't bother, as the sensor readout noise, should be contained within the darks for the lights, and the darks for the flats.

Regards,
Humayun

[avandonk]

Could not have said it better Hamayun
bert

[leon]

I do it right, but this is just to much for me, :shrug: come on guys we are just taking piccies, ;) at my age I',m lucky to do that, and I enjoy it, lets just keep doing that. :thumbsup: :thumbsup: we are delving into much technology and losing the plot, it is not a compition who can out do each other

Leon :thumbsup:

[[1ponders]]

It's all horses for courses Leon. On most nights for pretty picture stuff with a DSLR, unless the data is looking good coming off the camera I'll slap together some flats and just play processing skills. But if the data is good then it definitely pays to go that extra mile. For more exacting work (photometry - still playing when the skies fine) using say SBIG then I'll go the whole hog.

[leon]

Paul, I do go that extra mile, I take ICNR, I take flate's with each new session, I process as best I know how, but I do feel that sometimes, that it just gets to critical, and it would turn off new people, like I was once.

leon

[Tandum]

There is no way my flats can be the same exposure time as my darks. My darks need to be the same as my lights so how do I take a 20 minute flat, you can't. I think you meant to say darks need to be the same as your lights or darks and flats shold have the same F setting. I generally shoot flats to show about 30% on a PS hystogram. ie: 30% exposed.

Just to get it right :-
Light = an exposure of an object. (lenscap off)
Dark = an exposure of nothing. (lenscap on)
Flat = an exposure of a constant uniform light source. (lightbox on)
Bias = A quick dark exposure. (lenscap on)

Yes my images link isn't working, I'm moving it to a cheaper host.

[iceman]

Robin he's referring to darks for the flats.

So if your "flat light" is a 1 sec exposure, then you should be taking a "flat dark" of 1 second as well, and subtract them from your flat lights before applying the flats to your lights.

I don't take flat darks at this stage - my flats are usually 1/3s exposure so I didn't think it was worth it. I'll give it a try next time and see if it makes a difference (if I can force the camera to do ICNR on such short exposures).

Leon I know what you're saying, but sometimes it's the small things that make a big difference. I can tell immediately (usually) when an image hasn't had good flats applied. It's almost essential these days.

If subtracting darks from those flats makes the final image even 5% better for some people it will be worth it. Especially if it only adds a few minutes to their workflow.

[[1ponders]]

Thanks Mike. Yep that is what i mean Robin. When I take flats I take them manually with no automatic exposure so my flats can often be 3,4 or even 10 sec+ long. While not critical for DSLR shooting, if I'm doing Photometry with the SBIG then it is pretty much mandatory for some of the programs that are used. If the Bias has been removed from the light by subtracting a dark then the bias (and noise) needs to be removed from the flat to prevent that from being divided into the light.

It's often thought that flats are used to correct for vignetting, dust and other optical faults. This is true, but it's not the only thing they are used for. They're main function, particularly in 'sensitive' work it to equalize out the quantum inefficiencies found in the pixels across the chip. If my chip has a QE of 30%, that is only an average of the pixels. The actual pixels may well vary by a couple of percent. In that situation the flat helps to bring the pixels up the to same level, so to speak. This is why the bias should be removed. The noise in the flats will also effect the predicted QE so it should be removed as well.

Another issue is ISO. There has been a trend recently to use a lower ISO for taking flats because they are 'cleaner'. Changing ISO is basically changing the gain. for optimum flats I believe the ISO shouldn't be changed. If the gain is one then each electron is converted to an ADU so a pixel with 1013 electrons will read 1013 adu. but if the gain is reduced (lower ISO) and the gain is 20 electrons/ adu then that pixel will read only 50 adu, the other 13 being discarded. By dividing this flat into my image I am getting an incorrect adjustment because of the discarded 13 electrons. I hope this is making sense.

[Geoff45]

Of course if you are really fanatical and if you are using a monochrome camera with colour filters, then you should take flats for each filter separately as the transmission characteristics of the filters may neither be uniform nor the same as each other.

[avandonk]

If you use ICNR for your flats you are already subtracting a dark in camera if the exposure is one second or more.

Before I built the fridge I always used to use ICNR for flats and lights. With the camera inside the fridge and the ICNR off for collecting data (lights) it is impractical to switch the ICNR on just for the flats.

I got a very obvious banding pattern in one of my HDR images and it was coming from even seven second exposures at 200 ISO to get the trap in M42! There is no noise in a seven second exposure at 200ISO is a very wrong assumption!

It took me a while to figure out what was going on. I thought I would pass on the info.

Below are a couple of images showing the effect. One is a crop from the top left corner.

Bert

[Tandum]

Ahhh, so that would be the dark flat files in Deep SKy Stacker. :thumbsup:

[avandonk]

Paul you are quite correct. When dealing with any sensor all corrections should be done at the same level. That means same gain (ISO) etc.

I will investigate this further.

Bert

[iceman]

hmm if I kept my flats at ISO800, the actual exposure time would be very very short. It's already 1/3s at ISO100 with a lightbox.

I thought ICNR only kicked in when the exposures were longer than that. I'll have to check it out.

[avandonk]

In your case Mike it is just a matter of subtracting the bias frame from your very short exposure flats. A bias frame is the shortest exposure your camera can produce. Just take a dark at the same exposure and conditions. ICNR only kicks in at one second exposure or above.

If you use ImagesPlus then this is a simple procedure.

Bert

[iceman]

ok i'll try that. Thanks Bert.

[avandonk]

Leon it is not a competition as I would not tell all and sundry how to do it better if it was. It is about sharing info to get better results.

Astrophotography is the most difficult pastime ever invented due to all the variables.

I really enjoy images produced by top end equipment by others. My philosophy is to do the best you can with what you have now. I thought Hubble would stop amateurs producing images. How wrong I was!

I am figuring out ways to overcome light pollution and air pollution and still persevere.

Bert

[Quark]

Thanks for starting this most interesting thread Bert.

As someone only just getting into DSLR imaging I think this subject is very pertinent.

I wonder whether Mike might make this thread a "sticky" so that this information is easier to access and doesn't get lost as other posts push it further down through the forum.

Regards
Trevor

[Lester]

As Trevor said; thanks Bert for this thread.

I am not doing flats yet, but will soon and this informations will be very helpfull.

[Geoff45]

Here ia a helpful web site
http://www.hiddenloft.com/notes/CCDAP02.pdf

[Hagar]

I must be missing something in this as I understood a normal image process included the use of Darks, Flats and Bias files. Maybe I am wrong but it sounds like how to suck eggs to me.
Perhaps we can expect some images with a bit more colour and less washed out.

[Octane]

Doug,

Not everyone does the whole rigmorale. I can understand that.

I think the point of this thread was to show how important it is to subtract dark frames from your flat frames. Seems like common sense, but, a lot of people don't do it, and it shows in the final image.

Regards,
Humayun

[leon]

Humayun, yep I do it to the letter as well, just as I was showen by some of the very experienced on this site.

However I reckon where it all falls down is in the final processing of the data, at which some of us are not that experienced, me for one.


I know that if I could master all the aspects of processing the final image, my images, (although I'm happy with the results), would have more impact.

Leon :thumbsup:

[iceman]

Doug, the purpose of this thread was specifically about "darks for your flats". Not darks, or flats, or biases on their own.

Not everyone takes darks for their flats.

[Octane]

Leon,

I've watched your progression from your very earliest efforts to the newer widefields with the Tak. You've come a very, very long way, in a relatively short amount of time. :)

I'm glad to read that you do the whole process! :)

You're right about the final processing. It's something that takes a long time to master. Where most people probably fell asleep reading Jase's very descriptive image descriptions, I'd soak that stuff up and love it.

Regards,
Humayun

[Hagar]

Mike , with flat framed exposure times normally held well below the 30 second mark there is absolutely no need to take darks for your flat frames. Bias files on the other hand are quite a diferent matter. The read noise must be calculated into the flat frame to ensure a good quality master flat frame. The fact that most if not all image calibration programs go through a calibration and normalisation process when creating a master Flat and sets all the pixel integer values at a scaled value to set the calibration masterfile, the need for a dark is totally negated.
Berts example image is a section of an image which could well be a read aberation or noise or could be a harmonic setup by the fact that his camera is surrounded by TEC coolers etc which it was not designed to compensate for. Astro cooled CCD cameras have any aberation or harmonics created by electronics built into the in camera processing.
It must also remember that a DSLR while manufactured to work through a wide temprature range are optimised to operate correctly at a mid range area and to change this and try to reduce the operating temprature to the bottom end of the range will affect the operation of the camera in some way or another. Even the Astro DSLR's manufactured by central ds in korea, only cool the CCD and leave the remainder of the electronics alone. The cooling system of these is designed as a specific modification and is fully filtered to reduce harmonics etc.

Any dark frame calibration is taken into account during the overall calibration process and when the final calibration is staged in the software a combined calibration is carried out by a set routine which compensates for all of these abnormalities.
Flat frame images are not just a matter of taking a short image where the peak ADU count is 1/3 of the total full well capacity of the camera but should actually be taken at an exposure level which places the exposure in the linear range of the affected camera and all CCD's/CMOS sensors have a diferent linear range. To complete these files effectively you should graph the performance of your camera output and use an exposure which puts your flat exposure in the linear range of the camera sensor.
It is all a lot more complicated than just using an ADU count to guage your exposure.

Anyone who takes an image should use Darks Flats and Bias frames to calibrate their images and I know full well that a lot choose not to use some or even all of these calibration files. I am guilty of this on numerous occasions and usually nmention this fact when I post an image and usually get a comment back to ask why not. This choise is a personal one usually made for various reasons, be it time or just pure laziness, it is just a desision made at the time. If all images were to be used for full scientific purposes almost all images produced and posted on this and other forums would not cut the mustard.

The fact that this operation has made a diference to Berts image is great for Bert but may well impart other imperfections to an image when not using a modification like Bert has made to cool his camera to a point where at times may well be operating outside it's normal operating range. Any electronic equipment operating outside the normal will create small aberations which require further processing to repair. A top of the line camera such as the 5D should not be creating the type of aberation which is present in the attached image and may require some adjustment at the factory to correct this or it could be caused by something external to the camera.

Over compensation during the calibration process can and does lead to images being washed out and devoid of depth of field and is quite evident in a lot of wide field images in particular. Images often appear to have a grey wash or appear to have been taken through fine cloud which even strips a lot of the colour depth out of the image.

[avandonk]

Doug makes a lot of valid points. I am happy for him to be devils advocate as I do not possess all knowledge. The banding pattern and noise you see in the dark for the flat are at room temperature. This banding pattern and noise is far higher than any banding pattern or noise in any images taken with the camera at -10C. What was happening was that a really clean image was being unduly modulated by the uncorrected flat. In other words the banding pattern and noise was being introduced by the flat correction.

I tried using bias frames for correction in IP but these were taken at -10C.

I have found it is far better to use a dark frame at the exact same temperature as the lights. And use a flat corrected with a dark at the same temperature whether at ambient or -10C. Then no bias frame is needed at all in the corrections as it has all ready been taken into account.

The main purpose of the fridge is to have the camera in a constant temperature environment. The fridge is temperature controlled to plus or minus 0.1C even with large ambient temperature fluctuations that we get in Melbourne over one night.

On a hot summers night say 22C I run the fridge at -5.0C. On hotter nights even higher. These temperatures are well within the operating range of the camera as the electronics and sensor will be several degrees hotter than the fridge temperature. In fact the sensor rises by about two degrees from idle (camera on) to taking long exposures. So it is important to equilibrate the whole system by having the camera constantly exposing so then the darks will match the lights exactly. Even then it is not quite as good as ICNR! The real advantage is I get twice the number of images per night.

Healthy criticism should not be discouraged. We can all learn something from each other. I wonder where Jase goes for advice?


Bert

[leon]

Humayun, many thanks for your encouragement much appreciated, ;) as for Doug and Bert, :shrug: excellent points to ponder on, but I might just stay out of this one. :whistle:

Leon :thumbsup:

[avandonk]

Here is an image done with a corrected flat and showing full frame.

Full res 3.8MB
http://members.optusnet.com.au/~fmlee/CarNF.jpg

Here is the same data with an uncorrected flat.

http://members.optusnet.com.au/~cheekyfish/carhdrD.jpg

The really faint data is far better.


Bert

[Terry B]

The entire crux of this debate is about reducing the noise in an image.
Noise comes from various sources including (but not limited to) dark noise, read noise, random noise, cosmic rays etc.
The important 2 here are probably read noise and dark noise and it depends which is more prevalent in the exposure.
For short exposures the read noise becomes greater than the dark noise so subtracting a dark that has inherent read noise in it may actually increase the noise in the image. This is when just using an offset for the flat frames is of benefit.
You can reduce the read noise by averaging large numbers of dark frames or bias frames before subtracting them. This is where ICNR falls down for short exposures (and probably why the cameras don't allow it below a certain exposure time.)
If the flat frame exposure is long enough that the dark current is the dominant source of noise then the subtraction of a dark will help.
In the end for non science images it probably doesn't make a lot of difference.
I've attached an article about the detection of exoplanet transits using amateur equipment. It has a very nice explanation of the sources of noise in images.

[Hagar]

Do I read the title of the second file correctly and guess it is a HDR processed file or is it a true corrected or uncorrected raw image with just a possible linear or curved stretch?

With both images could you also give some idea of the actual calibration used. eg. Flats, bias and darks or Flats, flat darks, darks etc. It will make understanding the diference a little easier.

Thanks