DSLR RAW data calibration/preprocessing - some helpful findings - possibly?

[rcheshire (Rowland)]

I thought I would share this. Perhaps it may be of help to DSLR imagers. Following discussion and assistance on another forum, I am much happier with results, revisiting and reprocessing old data. More of a query, really? Thoughts, experience...?

Taking an astro processing software neutral position, having tried this on two prominent preprocessing packages, reading time and again about issues with another "free program," DSLR RAW data and particularly Canon, in Craig Stark's own words, "is mangled." I hesitate to say, that what you get after all the magic is not necessarily the best result, though it may look OK.

You may be aware that RAW data is subject to camera firmware manipulation, such that it is not strictly linear and not ideal for scaling and application of processes usually applied to linear data - which is a default for some preprocessing software, but changeable.

I have spent quite a bit of time in recent weeks working with RAW data from an uncooled 5DMKII and Fuji X Pro1 and a temperature regulated, cooled 1000D. The Fuji, from what I can tell, using RawDigger, is not so mangled. The Canons hot or cold are not as consistent.

Regulated cooled DSLR data is remarkably consistent, but this falls apart where it is assumed that the data is necessarily linear. The adage "the bias is in the dark" holds true for DSLR data, which is better processed from that perspective, cooled or uncooled, from what I can tell? Bias frames or dark flats are required for flat calibration - bias and flats are nearly linear, as opposed to darks and lights. In a nutshell, don't calibrate the darks.

The real issue behind not calibrating darks is, dark and light DSLR data linearity is unreliable, however, both share approximately the same bias and dark current (other things being equal - exposure time etc.), from what I can tell? Therefore, it is "safer" to calibrate light frames with an uncalibrated master dark and forget about scaling.

Of all the combinations, with 3 sets of data, particularly the Canons (exclude supercooled DSLRs, -25C, if you have one), calibration of light frames with an uncalibrated master dark frame produced the best results. Even at -5C, bias only calibration hurt my light frames - go figure? I conclude that the most reliable and complete calibration applied to DSLR data is had with uncalibrated darks, even though at -5C it's difficult to tell which is a bias and which is a dark. Flats however, can be safely calibrated with a master bias.

If this is useful or someone has a sure fire way of scaling DSLR data, while retaining most of it, following preprocessing, I would love to hear from you.

[Octane (Humayun)]

I never once used bias frames when I started imaging all those years ago. My results showed that they weren't necessary, too.

Because the bias is included in the darks for the flats, why are they necessary for flat calibration?

H

[rcheshire (Rowland)]

That's a good point H. I guess it depends on how the software works as far as flats are concerned - open to suggestions on that one.

I think you are right, however you arrived at a decision not to use calibrate darks.

I might rephrase the section about bias in the dark. It sounds critical, but not meant that way. I would gladly dicuss my frustration with that off-line.

EDIT: Do you mean use dark flats or just, no bias frames for flats?

[Octane (Humayun)]

Hey mate,

When I used to DSLR image (and, every now and then, still do), my captured data was as follows:

lights (as many as possible);
darks (usually about 16 or 25; with a third taken before I begin, a third during the middle, and a third after I finish -- this captures the range of temperature spread throughout the session and makes for a decent master dark);
flat lights (between 16 or 25); and,
flat darks (between 16 or 25).

No bias frames to speak of, as my understanding was that the bias was always included in the dark.

The flat frames are typically in the order of 1/30th to 1 second. I might have to take some bias frames the next time I do some DSLR imaging and then process the data with and without bias frames to see what the final output image (before Photoshop) looks like, noise- and calibration-wise.

Cheers.

H

[rcheshire (Rowland)]

I think that's the best approach without DSLR cooling. For cooled cameras the only change would be to create a separate temperature/iso/duration dark library. I have a suspicion that darks taken at the same shutter speed as flats is adequate for flat calibration - that is, dark flats. These are easily processed as bias frames in programs such as PIs BPP.

My workflow for any program would be 1. create master bias (moot point)/dark flat; 2. flat calibration and master flat creation; 3. create master dark (no calibration); 4. Calibrate lights with master dark and master flat; 5. DeBayer, align etc.,

I think this better preserves DSLR data integrity, given the relative non-linearity of DSLR RAW data.

[JohnG (John)]

Hey Roland

There has been quite a bit of discussion on the ImagesPlus group lately about Dark Flats, the opinion of Mike Unsold is that there is no need for Dark Flats in exposures less than a second or so (DSLR).

Cheers

[Octane (Humayun)]

Hey John,

Say if you're using an older model camera, and running it at ISO-1600 or ISO-3200, and take a flat of 1 second. If you take a corresponding dark, you will easily see that it is riddled with noise.

Wouldn't that noise accumulate in the flats, particularly if not many were taken?

Cheers.

H

[JohnG (John)]

Not sure H, I have always used Dark Flats although on the 60Da I haven't. There was quite a lot of posts on the group concerning Dark Flats and Mike Unsold was adamant that on exposures under a second or so there was no need for them, I learned under the really old system and still take Bias, Darks, Dark Flats and Flats.

Will give the not taking of Dark Flats and Bias a go next time out with the 60Da.

Cheers

[Octane (Humayun)]

Would be interested in your results!

I'll try the same next new Moon period.

H

[Asterix2020 (Paul)]

Another thing I have been doing is using dcraw to convert my RAWs to TIFFs before stacking in DSS as recommended by Ivo (thanks Ivo). This has increased the dynamic range in the final processed image.

I've been confused about bias and dark flats. Some people saw that Canon dark frames already have a bias in them so bias frames are not needed. Some say don't use dark flats and bias as the read? (not sure if read noise is the correct term) noise is subtracted twice, so only use one or the other. But if dark frames already have the bias, would using dark flats subtract the read noise twice? And if my flats are in the 1/80 second range, would dark flats be needed?

Also, is there any reason not to use lots of dark frames if you have them? I set the camera up last night and could use 100 temperature matched dark frames for an image. Since I have them, is then any harm in using them? I understand there may not be a lot of extra improvement over say 50 frames, but would it make the image worse?

I'm using a CDS-600D (cooled, modded Canon 600D).

Thanks for starting this, I think I'm going to learn a lot.

[rcheshire (Rowland)]

Hi John. Good to hear from you. I guess there is no dark noise to speak of at fast shutter speeds, anyway. I wonder if there is a benefit to DSLR RAW data presevation, keeping exposure times matched for calibration frames; that is, lights and darks, and flats and bias at the same shutter speed (flatbias, for want of a name).

[JohnG (John)]

H.

Just to clarify, I take my Flats using AV Mode on the camera with a Light Box so the camera automatically sets the Exposure, with high ISO Subs I believe you can set the ISO on the AV Mode a lot lower as that is not an influence on final image therefore your actual Flats can be taken at a relatively fast exposure speed.

My normal working ISO is 400 to 800 anyway. Everything is taken in RAW.

Cheers

[rcheshire (Rowland)]

Originally Posted by Asterix2020

Another thing I have been doing is using dcraw to convert my RAWs to TIFFs before stacking in DSS as recommended by Ivo (thanks Ivo). This has increased the dynamic range in the final processed image.

I've been confused about bias and dark flats. Some people saw that Canon dark frames already have a bias in them so bias frames are not needed. Some say don't use dark flats and bias as the read? (not sure if read noise is the correct term) noise is subtracted twice, so only use one or the other. But if dark frames already have the bias, would using dark flats subtract the read noise twice? And if my flats are in the 1/80 second range, would dark flats be needed?

Also, is there any reason not to use lots of dark frames if you have them? I set the camera up last night and could use 100 temperature matched dark frames for an image. Since I have them, is then any harm in using them? I understand there may not be a lot of extra improvement over say 50 frames, but would it make the image worse?

I'm using a CDS-600D (cooled, modded Canon 600D).

Thanks for starting this, I think I'm going to learn a lot.

To answer briefly Paul. I like the idea of dcraw conversion applying optimal options. I read that post. Secondly, if you adhere to bias in the dark to calibrate your lights and don't use bias at all, there are no double subtraction issues. As to flats, calibrate them separately with a bias or as in my previous post, below. This might mean double handling to produce master dark and master flat frames, but it's worth the effort.

EDIT. Lots of dark frames produces a better mean, afaik. EDIT: See up thread.

[JohnG (John)]

Roland

The convention as I believe is that your Subs and Darks will always be the same length exposure, ISO and temperature as you know. Like H, I try and take some Darks at the beginning, then the middle and at the end to get an even spread of temperature.

I am not into the physics of how it is done, just know it works OK for me , I have always been lead to believe that the ISO of Flats is of no consequence so long as they are the same size as the rest of all the frames. I suppose when you think about it the fast shutter speed of a Flat may well negate the use of a Dark Flat....

[Octane (Humayun)]

Using the lowest (native) ISO of your camera for taking flat lights and flat darks would mean they would have the lowest noise possible in them. Everything contributes in one way or another. May as well minimise it wherever we can!

H

[rcheshire (Rowland)]

Data set would look like this;

1. Lights
2. Darks (no bias subtraction) == lights
3. Flats - low iso
4. Bias, dark flats, however described (low iso) to calibrate flats only - alternatively to keep things equal take so-called bias at same speed as flats, as mentioned previously.

That sounds like a collaborative effort. Thanks guys. I forgot about low iso flats. I think it means double handling, however the flats are created which is unavoidable. Create masters separately and use to calibrate lights.

EDIT. I like the idea of intermediate dark frames.

[Merlin66 (Ken)]

AstroArt5 recommend using a Flat Dark - a dark taken at the same setting as the Flat....this then means you don't need any Bias images etc.

[JohnG (John)]

Roland

If you are going to the next Snake Valley Astro Camp we can sit down and have a talk about this, I now have the FS-60 taking images with pin point stars edge to edge on an APS-C chip so it will be my field setup with the Canon 60Da.

An interesting thread .

Cheers

[rcheshire (Rowland)]

Thanks Ken. It seems we are on the same track - what a relief. Perhaps the term dark-flat can be applied generically in this case, forgetting exposure time.

John. That will be good. I look forward to it. Pinpoint stars with an FS60CB. It might be time to retire the 200mm Canon.

Rowland.

[Amaranthus (Barry)]

Lots of dark frames produces a better mean, afaik

The standard error of the mean is inversely proportional to the square root of the sample size, so it is a matter of diminishing returns.