Digital Imaging - Flats and Darks Explained!

[iceman]

Eddie Trimarchi, one of Australia's best astrophotographers, has kindly written a how-to guide for the use of Flats and Darks in digital imaging.

You can find the article at the IceInSpace How-To (http://www.iceinspace.com.au/?howto) page, or by clicking on the following link:

Digital Imaging - Flats and Darks Explained! (http://www.iceinspace.com.au/index.php?id=63,211,0,0,1,0)

Many thanks to Eddie for contributing such a great article, and sharing his knowledge so that others can benefit and produce better images as a result.

You can see Eddie's great astrophotography at his website: The Tin Shed Observatory (http://www.astroshed.com/)

[[1ponders]]

Thanks heaps Eddie. At least now I understand what it is I'm trying to do and why. You make it sound relatively simple. Now if I can just understand how to get the imaging programs to do what you've suggested I'll be laughing.

I love the idea of the flat box. I've tried to do a few off light coloured wall but I've not been able to get the illumination even enough. Great article.

One thing I'd like to see as an adjunct to this article (and I'm not trying to put you back to work Eddie, you've done a fantastic job :) Anyone who is able and has the knowledge could do this ), is some additions to the "Iceinspace Glossary of Terms" explaining some of the terms you've used, like Normalization, Registration, Optimization, median combining etc. This could also apply to any other article that is posted that uses specialized jargon to explain the topic.

Mike is there anyway the articles could have a link to a "printer friendly version". I'd like to be able to print articles like these for "in field referencing" and currently it's take 8 pages to print.

[iceman]

Paul, there is a "Print" icon at the bottom of the article, which makes it a print-friendly version. It's a long article, I don't understand what you mean by "printer friendly" and trying to fit it in less than 8 pages?

If there's only some sections you want to print (to take into the field), just copy it into word and mangle it how you'd like.

[[1ponders]]

That's what I normally do Mike (ie take it to word). By "printer friendly" I mean when it comes up in print preview, its minus all the frames at the sides and expands the margins so that more info and less webpage layout appears in the printed version.

See here http://www.designplace.org/tutorials.php?page=1&c_id=27

[iceman]

You mean, like this? (http://www.iceinspace.com.au/index.php?id=63,211,1,0,1,0)

Just click the "print" icon at the bottom of the article (not the standard IE print toolbar icon).

[iceman]

oh don't go and delete your post, it looks like i'm talking to myself! :)

[[1ponders]]

But that's how I get my best conversations :P OK I'll move it back if you want:) Yes that's what I'm looking for. I guess if I really want to use as much of the page as possible I'll take it into word.

Ok I'll be quiet now and get back on topic.

[h0ughy]

At least Mike you get the answers you want :D

[EddieT]

You're very welcome!



Yes indeed. I tried to keep it simple and introductory focusing mainly on the reasons for image calibration (with examples), without specific step-by-step instructions, because these will vary from application-to-applicaton. I think it's better to know what you are trying to acheive and then finding out how to acheive it with your specific spftware.



Making a good flat field is paramount. It doesn't take much of an inconsistency to create an inaccurate flat, and applying a bad flat can make your images worse than before. Much worse in fact.


This was constantly in my mind as I wrote the article. But whenever I tried to concisely explain a statement that needed clarification, the article grew in size exponentially :) I had to draw the line somewhere. In doing that, I decided that certain terms would have to be resourced elsewhere and this article would be the catalyst for that "further research". It was that or end up writing an online book!

I am open to questions regarding any part of the article so if anyone needs help to understand something, please ask. This will also give me a better idea of which parts are unclear and also framework for an addendum if needed, or a future, deeper article.

[[1ponders]]

I'm working on that. :D

[Geoff45]

I'm just finishing off a light box now. One question: does anybody know if the color cast of the lights in the box is important? It seems to me that if you are subtracting a flat from a colour image, the effect may be different for different lamp colours.

[EddieT]

Hi Geoff,
Going for bulbs as white as possible is safest. The spectral response of your camera/filters will have an effect on what colour the flats end up. Most one-shot cameras aim at a white balance that shows white as white. i.e. even amounts of R,G and B where the colour is white. The interior of the box should also be white as well as the diffuser. The aim being to get an evenly distributed flat white light source to calibrate with.

[gbeal]

Eddie,
I saw this article when you first floated it, and was impressed. I have mastered darks, but really I struggle with flats, silly I know, but I do.
I have decided to rattle out a light box like you suggest, and may come back for a little more fatherly guidance.
One thing I did wonder, is any specific suggestions for bulbs. I too am on a one shot colour CCD, so this is equally important for me. A suggestion from somewhere like DSE (which we have here as well) would be appreciated.
The flat wall flat isn't really as good as I had hoped.
Regards,
Gary

[Doug]

I have been using a white cartridge paper over the dew cap with the OT aimed at the zenith. Colour balance on the calibrated image is done using a G2V star. I think this should compensate for any white imbalance in the paper.
Eddie's light box has some attractive advantages over a sky flat. Not sure how white a low level light source is, however a G2V star balancing act should take care of any colour bias introduced by the flat field image in the final calibrated image.

[Geoff45]

What happens if you just desaturate the flat and get a grey image?

[[1ponders]]

I've been using standard 20W 12V halogens (couldn't get the ultra whites :shrug: ) and the box works a treat. What I did have to do though was hook the bulbs up in parallel (as suggested elsewhere) and install a dimmer switch on the 240V side of the transformer. Does a great job.

[Doug]

You don't convert the flat to colour Geoff; it is a calibration frame. My reasoning, FWIW is that if there exists within the flat frame any bias toward one or two of the primary colours. that will translate as a colour caste to the final colour image. By having a G2V raw and calibrating that, when it is susequently colour converted, and balanced, the same colour values will give a correctly balance image of whatever it is you are imaging. The G2V is used as a reference colour, and adjusting the colour scaling at the time of colour conversion such that the resulting image of the G2V star has equal values of RGB, should therefore result in correct colour in subsequent images calibrated with the same set of flat frames.

[Doug]

20W:eyepop:, what sort of exposure times do you use Paul?
And how do you position the box? Sit it atop the OTA, or on a bench at a distance? I have not ventured into the realm of light boxes, one thing that worries me is introducing an artificial gradient. I think I would be happy with a dim enough light source such that the shutter/blind travel time is an insignificant % of the overall exposure time. The other concern;real or imagined is if the light box diffuser surface is not perpendicular to the optical axis, this could lead to an unwanted gradient. Been thinking of trying it out though because it can be a real pain setting up in daytime. With the flat gathering method I use I've found it best to use a mid day(ish) sky rather that twighlight (contrary to the words of the Gurus) because of the rapid change of light levels around dusk or dawn.

[[1ponders]]

I just followed Eddies lightbox design but with the bulbs in parallel (I had problems with them in series as in the design) and put a dimmer switch inline on the ac side of the transformer.

Yes I just invert the ota and put the box on the top. It is very light and a snug fit so it doesn't fall off. The diffuser screen I use is perspex that is used in the lighting industry for that particular purpose.

Here are a few piccies and associated data.
The first one is a shot of the finished box.

The next three are a shot just using a lense in a dark room with the box at full setting (very bright) with it's colour histogram and the raw camera data.

The next three are of a shot taken through my 8" with the 300D with a colour histogram from PS and the Raw image data.

The last is of the box sitting on the scope. Works just as well on the ED80.

It should be noted that I checked the consistancy of illumination using a light meter and there was less than 1% difference in reading anywhere over the box. Any gradient that may be seen in the image is an artifact or produced by the scope or lens or response from the camera

btw the box has now been fully taped around the edges. When I took the shot I thought I may have needed to still get into the box to make adjustments.

[[1ponders]]

Remember too that the raw Canon images are only 12 bit. When I convert them to 16bit fits files any variation in illumination differences is increased. Plus the accuracy of the light meter is an unknown with the responsiveness centered on the green. It would be interesting to split the colours and just check the green channel.

I did do a conversion (colour to fits without splitting the channels) and using the image of the whole box illumination and checking the mean value at various points there was a less than 6% variation in intensity from the edge to the center.

[Doug]

The quality of light used for a flat field certainly seems to affect one shot colour. I took a series of flats today with a blue sky lighting through white cartrige paper and the results are noticably different from flats taken with an overcast sky as the light source.
I think it is time to consider building a light box. Jerry Lodregus seems to think LEDs are not a good idea, but doesn't seem to say why.
I used to service a certain negative film scanner that used arrays of red, green and blue LEDs to illuminate the negs. quite impressive display of colour as it flashed thru the colours, varying time intervals to give a white balanced exposure.
Of course in that application there was no need to try to produce white light, as the beastie just read the three channels and went away and did its homework. Still, I wonder if a Light box could be worked with a set of LEDs tuned to give white light?

[gbeal]

I haven't done anything re lights, but now that the new refractor has arrived (and is likely to be THE imaging scope) I need to.
I'll check this end, but any suggestions on type/brand etc are appreciated.

[netwolf]

Great article Eddie, makes some sense now what the DSO guys are on about.

Can someone however explain what a bias frame is and how it is different from a Dark frame?

Regards.

[[1ponders]]

A bias frame records the random noise contributions from the sensor's on-chip amplifier. To take one, you set your camera to the shortest exposure it can take.

A dark noise is a measure of the noise generated by the chip over the timeframe of a light exposure. However when you take a dark frame you are including a bias measurement anyway. Depending on what you are doing with the image then just a dark frame may be all that is necessary without taking a seperate bias frame.

[Doug]

Well yes, a bias frame is only necessary if you want to scale the dark frames.
Because a bias frame is fundamentally part of a dark frame, it becomes attractive to be able to subtract its influence prior to multiplying or dividing the time of a dark frame. In other words if I want to use a 5min dark frame on a 7min light frame, multiplying the dark by 1.4 will not be the same as taking a dark at 7mins because the bias value would also have been multiplied by 1.4. So scaling will sutract a bias frame from the dark, multiply/divide as required and add the bias value back to the resulting scaled dark value. So Bias frames are only required for scaling work.

[Rigel003]

Great article. Thanks, Eddie.

I'm pretty much a beginner with darks and flats with my DSLR and still have lots of queries. The discussion about the quality of light for flats is interesting. I've been doing T shirt flats with the scope pointed at a wall illuminated by a tungsten bulb. I was worried that these had a pinkish colour cast and wondered if I should change them to grayscale before using them, but Mike Unsold said that it didn't matter when the automatic image processing routine in Images Plus dealt directly with the RAW frames. Does this mean the nature of the light source doesn't matter?

Also, Jerry Lodriguss seems to be saying that if you keep your light source sufficiently bright that the flat exposure can be under 1 second with a low ISO, then it's not necessary to take extra dark frames for the flats. I hope this is so because an extra set of darks for the flats seems like one step too many. Life is too short.

[darrellx]

Hi All

In this article, the statement "The main aim is to get a flat field that has an average pixel value of about 30% of the maximum pixel value that your camera is capable of. For a 16-bit camera, this would be approximately 20,000. Most image processing software will give you the average pixel value in an image so this value is generally easily obtained" has me a bit stumped.

I use CS3 and PaintShop Pro, and I just cannot find anywhere in there a place to find the average pixel value. I must be missing something.

Can someone point me oin the right direction to find this "average pixel value".

Thanks
Darrell

[Doug]

G'day Darrell,
Mostly I suppose people have dedicated asto-imaging software in mind such as images plus, maxdslr etc.
If you are using a DSLR and saving in BMP etc. you might try playing around with the expanded histogram window in photoshop to find the mean pixel value. If you are in a position to save in FIT(s) format, you will most likely find a useful readout using fits liberator, as a photoshop plug in.
What are you imaging with? A lot of capture s/w can also calculate the average...well some programs anyway.
cheers,
Doug

[darrellx]

Doug

I have an EOS400, and usually save my shots as jpeg and raw. With a typical photo, I use the histogram, and for the work I do on the photos I am comfortable with this (at the moment).

So I have been reading about flats and thought I should give it a go - I currently just do the darks. When I take the flats, there isn't anything in the histogram. Its really odd. There is just nothing there. In this article, mention is made of the average pixel value, so I thought I would look at that to see if I am on the right track.

Until the last month or two, I have used PaintShop Pro. But just before Christmas, I got CS3 and am still finding my way around.

Darrell

[EddieT]

Hi Darrell,

Obtaining the value in Photoshop could be considered to be a little "too late".

The idea is that you must achieve this value when capturing the flat field, so you really need to be able to read it from your camera control software after you capture the image.

You would adjust the exposure time to achieve your 30%. For example, if you take a flat and the average pixel value reads 10000, then you would need to roughly double the exposure time and retake the flat.

It all depends on the brightness of the light source being used to take the flat from.

Make sense ?

Eddie Trimarchi
http://astroshed.com

[darrellx]

Eddie

It does. Thanks. I have been taking a set of them; downloading to my laptop; checking them; then redoing it if neccessary.

So if I interpret all this correctly, I should either get the camera control on the laptop working, or check the histogram on the camera after the first shot and make neccessary adjustments.

I might give it another try tomorrow.

Darrell

[EddieT]

Hi again,
What are you capturing the images with ?

If it's Maxdslr (which I have no experience with), and it is the same as MaximDL, then choose View/Information Window (Ctrl-I).

Check the Aperture size (View Area) in the window, it should default to the entire window and it should show you the min, max and average pixel values. If it doesn't try minimising the view to 25% to fit the entire image on the window, then drag the cursor round the entire frame to highlight as much of it as possible. If nothing still shows up in the information window, then you have a problem, most likely with Maxim if you can see that the image actually has data in it, (indicated by varying brightness in the frame).

[darrellx]

Ah, sorry. Well, I don't actually use any software to capture the images. I just use the 400D and take a lot of frames. I attach the camera the way I take the photos - sometimes prime focus, sometimes with EP projection.

I just thought I could analyse the jpegs or raw files after I load them into CS3.

Darrell

[EddieT]

Hi Darrrell,
ah, I see now! Do people do that ? :)

Sorry mate, I'm so into my own headspace I never realised that you might not be using image capture software.

In Photoshop you should still be able to see the mean, I guess you can see the histogram (Menus: Window/Histogram) but unfortunately Photoshop only shows the screen histogram which is only 8-bit even if you have a 16-bit image loaded. In CS2 anyway, perhaps CS3 is more friendly.

You can still get an approximation of 30% maximum, only the mean will be 30% of 256, rather than your 16-bit value. Try going for a value of 65-70 in Photoshops 8-bit histogram and see how you go.

[darrellx]

Eddie, sorry for the confusion. I am still struggling a bit on the whole image processing thing. Until I started on this last year, I really did believe that all those amazing shots you seen on sites like this and in magazines where taken as a single exposure from an amazingly expensive camera. From that, I suppose the good news is that it need not be that expensive. The bad news is that it is much much more complex than I thought.

You are correct about CS3, the screen histogram is 8 bit. I also have a menu box at the right side of the screen with three tabs - Navigator/Histogram/Info. If I select the Info tab, I have values against R, G, and B, and also for C, M, Y, K. As I move the cursor around, the values change. So I figure that is the value for the pixel the cursor is on.

How do I see a value for the average of the entire image? The "mean". When I look at the Histogram tab, there is only the curve displayed - no values for the overall image, and no values on the x or y axis.

Thanks for your help.
Darrell

[darrellx]

Eddie
Ahah! Got it. In Measurement Log/Record Measurement there they are - all the measurements I could want.

Thanks for all you guidance. Now to put it to good use.

Darrell

[Terry B]

My 40D and I assume the 400D should show you the histogram on the LCD screen on the camera after you have taken the image. Just expose the flat to make the histogram in the middle.

[rogerg]

I might be a bit late on this conversation but better late than never:

If you did want to work out the average brightness of an image in photoshop this is how I would do it:

1) perform a gaussian blurr (filter menu -> blur -> gaussian blur) with an extremely high value, if not maximum value.

2) show the "info" toolbox (window -> show info, if not already visible). this shows mouse cursor x, y and RGB colour values as well as a few other things.

3) select a tool such as the eye-dropper or Move Tool.

4) move your mouse over the image, the areas of average brightness (if there is still any variation after the gaussian blur) and watch the RGB values in the info toolbox. They should be quite consistent. That will be your average brightness.

As others have said, they should be in the range of 60-70. If not, you need to increase or decrease your exposure times appropriately.

Roger.

[darrellx]

Guys

Thanks for the pointers. I have had a look at both - the histogram on the camera and the gaussian blur. I think I should be fine now. At least I sort of understand what I am doing.

Now, I might build myself a lightbox.

Darrell

[Quark]

Hi Eddie,

Have read your most excellent article and am in throws of building a light box, as per your basic design.

Can you provide any more info on the bulbs that you use. My local supplier assures me there is no such animal as an ultra white halogen globe, that halogen globes provide warm light ie: a yellowish light.

Any clarification would be greatly appreciated.
The box I am building is for a 16" scope so the diffuser will be quite large.

Regards
Trevor

[BerrieK]

Alas I feel that your local supplier is incorrect. I believe you can get tungsten or xenon ultra white halogens. Maybe try an auto shop - they are sometimes available for car headlights (depending on what fitting you need). They are also sometimes use din shops to really bring the true colours out in displays.

[Octane]

Trevor,

Do you really need halogen bulbs?

I used high bright LEDs in my lightbox (http://www.iceinspace.com.au/forum/showthread.php?t=38103) and they work a treat. I bought them from JayCar.

Regards,
Humayun

[Quark]

Thanks H,

Looking through this thread the only mention I recall of LEDs was that they may create a problem with one shot CCD's using various filters. I will only be using my 450D. I have used bright LEDs in the past to make an all round white anchor light for my boat. They were very bright, definitely white and consumed very frugal amounts of power.

It is most reassuring to know that you use LEDs, I really admire your work and have the greatest respect for your opinions on astro imaging.
I will now confidently go ahead and use LEDs.

Regards
Trevor

[gregbradley]

My understanding of bias frames is that you subtract them from your flats and then you do not need to use them again.

Also its best to use simple average for your flats and sigma reject for combining darks.

Greg.

[KenGee]

I was looking to use a white shirt over my telescope when using my newEL flat pannel light. But I don't own a white shirt. I did have a ceap canvas pannel from a art and craft shop that I had been using for my flats. I tried it out today it works well.

[RobF]

Don't know what the experts'll say Trevor, but I use these from Jaycar (http://www.jaycar.com.au/productView.asp?ID=SL2675&keywords=lamp&form=KEYWORD), as per the ASIGN design (http://www.asignobservatory.com/diy_do_it_yourself/light_box.aspx), and get pretty similar flats to Paul (albeit with much longer exposures - about 0.5sec). They have small and "large" lamps with pre attached wires, so soldering up a bit easier - large ones a nice size for me.

I had an old 17" dead LCD screen I pilfered for diffuser screens that seems to work nice too (saved hunting in specialised lighting stores)