When can you image effectively without flats?

[Shiraz]

Can you image effectively without using flats? Mike S. does so very effectively and some camera makers suggest that it can be a useful strategy. While trying to sort out a calibration software problem, I did some modelling that seems to indicate that imaging without flats can be a good idea, but not for everyone - like many things in this hobby, “it depends”….

As I see it, flats have 2 purposes:
1. To remove large-scale illumination defects such as vignetting and dust bunnies
2. To remove noise generated by sky radiation interacting with pixel-to-pixel CCD sensitivity non-uniformity (fixed pattern noise or FPN).

If you have clean fast optics and a smallish CCD, you may not have major illumination defects and can get by without using flats for purpose 1.

Efficient imaging is also possible under some circumstances without using flats to remove FPN (purpose 2). To put some numbers on the effects of FPN, the attached graphs show how the SNR varies with total integration time for a variety of CCD Non-Uniformity (NU) values (using a 10inch high-QE system system with about 1 arcsec sampling as an example). The solid lines show how the SNR increases with integration time if you use flats to get rid of FPN. The dashed lines show the SNR performance without using flats and for three different CCDs, with NU values of 0.2%, 0.5% and 1%. Non-uniformity data is a little hard to come by, but I think (hope) that these values represent the types of CCDs that we use.

With dark sky (left hand panel) and a CCD with low NU (0.2%), flats do not make much difference to the SNR, which remains fairly close to the values that would be obtained using flats. However, flats definitely do help with the other two CCDs under the dark sky, particularly for longer integration times.

With a relatively bright sky (right hand panel), flats are worthwhile for all of the CCDs and SNR is significantly impacted if they are not used. The situation is particularly bad for the CCD with 1% NU, where the FPN dominates (even for short exposures) and no amount of additional integration can improve the SNR.

It should also be noted that an NU of 1% is not really an indication of a poor quality CCD – FPN can be removed with appropriate flat fielding and then all of the CCDs will achieve the identical shot-noise dominated performance shown as a solid line in the graphs.

In summary, it seems that, if you have a fast scope, a smallish CCD with low NU and dark skies, then you can image efficiently without using flats. Otherwise, you need to use flats or you will be losing out on SNR, particularly over long integrations and under bright skies. Indeed, without flats, some CCDs will be practically unusable under a bright sky. Thanks for looking - would be grateful for any feedback/experience. Regards ray

Ref:
http://www.narrowbandimaging.com/incoming/flats_part1_part2_part3_expanded.pd f

[tonybarry]

Hi Ray,

May I ask how you would know you had no vignetting or dust bunnies without taking at least one flat ?

Regards,
Tony Barry

[ericwbenson]

Easy, take a long exposure image where the sky glow is significantly above the read noise, e.g. any properly exposed sub. The bgd pedestal will show vignetting, dust bunnies, and with enough bgd signal (more than just above rad noise) ccd non-uniformity.

In Maxim there is a graph tool to plot the intensity as a function of position for a line you draw on the image, it can be quite disturbing when used on an SCT + focal reducer how vignetting actually happens...it convinced me my C14 had a limited life span ;)

EB

[ericwbenson]

Hi Ray,
Nice graphs, well done. One thing not quite captured here, and I don't know how one would do it in a completely analytical way, is the fact that when we dither our images the FPN (I'll just call it that for now) is smeared when the stacking actually occurs. Does it make it worse or better, not sure, haven't thought that far. But I do know that it would be better if wasn't there to start with!

Point two: Many years ago I was shooting the breeze with Paul Boltwood and Doug George at Doug's place. Paul had just recently won the deep sky competition set out by Sky and Tel mag (amateurs imaging the Hubble deep field to see how deep they could go), and was quite surprised that he had beat others, since he imaged from an obs attached to his house, in suburbia (Kanata) of a ~750K population center (Ottawa). He had only a 16" newtonian reflector and a camera he had made himself (this guy was very bright and handy, I can't imaging designing and building a custom CCD holder myself!). One thing he said was: going deep without really really really good flats, fogget about it...flats ultimately are THE limiting factor in how deep you can go.

Best,
EB

PS. I just found a link to short bio on him:
http://astro-canada.ca/_en/a2223.php

[Shiraz]

Thanks very much for the info Eric.

I was also concerned about what happens when you dither. I think that it just randomises the fixed pattern noise and turns it into random noise like that from other sources - the noise power is the same, but the spatial distribution is no longer consistent with ref to the star pattern. That was the basis for the graphs - assuming random noise for the FPN made it easy to quadrature combine noise from different sources.

If you want to get as deep as possible, removing any manageable noise sources makes sense, so quality flats are essential. However, if your system has low non-uniformity, concentrating on lights and neglecting flats may be a perfectly sensible approach for many targets.

I was surprised at how much difference flats can make with some cameras.

regards ray

edit: that's an interesting site.

[RickS]

Hi Ray,

I definitely need flats with my KAF-16803. Interestingly, I take dusk and dawn flats every day when I have the scope set up in the back yard and I have found that having lots of flats does make a measurable improvement in SNR using PI noise analysis tools.

Richard Crisp did a flat field PTC analysis of one specific KAF-16803 and found that around half a million electrons was the sweet spot so that's what I usually target (around a dozen flats at 40K e- half well.) However, I recently discovered that if I use several times as many I can measure a significant SNR improvement. I must do some more PTCs myself when I get a chance. Maybe my sensor behaves differently to the one that Richard measured.

Cheers,
Rick.

[Shiraz]

hi Rick.

Interesting observation. I suspect that Richard's analysis assumes that the main FP noise is due to the interaction of the target signal with the CCD non-uniformity, whereas the big problem is actually FPN due to the much brighter sky. His noise equation (in the reference in an earlier post) has an FPN term of Signal*NU, so that is possibly the case. If so, you would need more accurate flats to compensate, since Sky*NU presents you with much more noise amplitude than Signal*NU. Will have to look a bit more closely at Richard's maths - I skimmed over a bit of his work :). Might send him an email.

Of course there is also the added complication of spectral variability - unless the spectrum of the flats matches that of the dark sky light, the flat calibration cannot be totally accurate, since the response of the sensor has variability with wavelength as well as intensity. Sky light at dusk will not be the same colour as dark sky light and could add noise due to pixel level spectral sensitivity variations. This is a real can of worms and I don't have any feel for how significant it is, but there may be a good case for using a white LED source for flats. Have you tried any alternative light sources for flats?

Regards Ray

[LightningNZ]

I take all my flats against an LCD screen with very short exposures so it never eats into my imaging time. Of course I can do this because I have a small scope (I also use a DSLR rather than a small-diagonal CCD chip so flats are pretty essential for me).

Dark and bias images seem to make very little difference to my images overall, but flats I consider to be pretty critical.

Sorry that my sample size of one is also an outlier for your thesis of small-area CCDs. :)

Cheers,
Cam

[Shiraz]

Thanks for the feedback Cam.

I am not trying to make a case in support of "no flats", just trying to show what circumstances are necessary before the technique can work well. If your optics produce vignetting over the chip scale, flats are essential, regardless of actual chip size.

Do you use a panel designed for taking flats and do you use flats to reduce vignetting etc, to reduce FPN - or both?

[strongmanmike]

And that's about it Ray...being able to do without flats is simply a function of ones system ie a fully illuminated field, a chip with low NU, clean correctly sized filters, wide image train openings to avoid vignetting, a dark sky and.. voila :thumbsup:

I am able to surface extremely faint features without flats and I have not seen a single dust donnut, not one, even with extreme stretching :)

Mike

[gregbradley]

Richard recommends taking around 20 flats. That's why if you are taking dawn/dusk flats you need a camera with fast downloads. Otherwise you don't have enough time to capture that many for each filter.

Greg.

[gregbradley]

Apart from dust bunnies I found flats on some faint galaxies seemed to make some images more noisy I had taken with 694/CDK17.

But I did not have 16 or so flats median combined either so perhaps that is worth pursuing before its a fact.

Dust bunnies require either a clean or flats. Such a small chip is a long way from being vignetted on that setup. Other scopes may vary. My 16803 totally needs flats on the same scope.

Greg.

[LightningNZ]

I realise, and I didn't mean to derail your thread - sorry if I have done.

I just use my PC's LCD screen for taking flats. With an APS-c sized chip on the AT65-EDQ there's no vignetting at all. I do get occasion dust marks on my images - I'm generally not too fussy about keeping the 2" filters spotless.

Cheers,
Cam

[Shiraz]

I have seen one dust bunny in the year I have used my 694 - not an issue with a fast scope and the camera window a fair way from the CCD. However, I have recently started using the dark/flat calibration routines in Pixinsight and am getting a useful increase in SNR from doing so - getting rid of some FPN can sometimes help.

Hi Greg - sorry that this overlaps with your previous thread to some extent. I also find that the 694 needs very good quality flats or else you can lose SNR by introducing noise via the flats - darned quiet cameras these are.


Thanks for the info Cam. Must try a laptop screen. I made up a foam box and that works OK, but it is a bit clumsy to use.

Threads tend to go where they will and you have in no way derailed this one - but even if you had, no problem :lol:

regards Ray

[Amaranthus]

I use the trick of covering the scope aperture with a white t-shirt, and then pointing at the inside of my garage (about 10 m away from my concrete pad) with the lights turned on. Seems to work well, and it means I can do them at night, right before or after my subs session.

[RickS]

Ray,

You could be right about the FPN and sky glow. I haven't really looked deeply into the maths yet either.

I do have a Gerd Neumann EL panel but I've pretty much stopped using it since I found that dawn/dusk flats were giving great results. If I get a chance I'll try a bunch of EL flats and see what I learn...

Cheers,
Rick.

[gregbradley]

I don't personally believe modern CCDs vary too much in sensitivity across the sensor.

Perhaps there is some variance with different coloured lights but again its not the main thing you are trying to correct.

Its usually dust bunnies and vignetting from your scope you are trying to correct.

Some scopes have hot spots/bright centres plus vignetting. This is becoming more common with the use of correctors on RC, Dall Kirkhams etc.

They become more of a challenge for flats I find.

I also find high quality darks and bias essential for a good flat correction on these types of scopes.

APOs are a piece of cake, its the ones with correctors that are tough.

Greg.

[Shiraz]

sounds like a practical approach if you have a nearby light source - thanks for the idea.


Hi Rick. Trying to get my head around it now :confused2: .


Hi Greg.

Agree, modern chips probably have less than 1% non-uniformity - for daylight imaging that is inconsequential. But for broadband astro, the main source of light is the sky background. If you have even very slight non-uniformity, it will be applied to the bright background sky light and then that bright noise will get mixed up with the much dimmer target. Even small sensitivity non-uniformity can introduce significant sky noise after you stretch to bring up the signal. eg if you have a sky that generates 1000 electrons in a sub and your target is much dimmer at only 10 electrons in a sub, then 1% of the sky will amount to 10 electrons - ie the sky non-uniformity noise will have as much power as the full the target signal - even minor non-uniformity is not inconsequential when you are trying to surface a dim signal buried under sky radiation.

The noise reduction action of flat calibration is in addition to the correction of vignetting and dust bunnies and it requires much higher quality flats. If you haven't already read it, discussed in http://www.narrowbandimaging.com/incoming/flats_part1_part2_part3_expanded.pd f

Regards ray

[Shiraz]

Hi Rick

Well that was interesting :). I went back to first principles and came up with what turned out to be a quite straightforward analysis and a simple rule of thumb for flat exposure:
if you want the flat induced noise to be below 10% of the total noise, then the total number of flat electrons must be 25x the total sky electrons in your lights. If you want it below 5%, you need 100x.

This is quite different to what I Understand of Richard's approach in that there is no single sweet spot for flats - if you expose longer, you need more flats - if your sky is brighter, you need more flats. Seems to make intuitive sense to me. Next step is to test it out in a model and then with a whole bunch of flats. When I have gone through the reasoning a few times, would be grateful if you would give it a once over before I post it (that is, if it holds up under testing - I still reserve the right to be wrong :lol:).

If this is correct though, the more flats the better and you will always gain something (even if small) by adding flat data. It also means that those who are forced to image under bright skies should stock up on lots of flats and that the "mega" brigade should also go mega on flats. And maybe I can get some better results out of the many sets of lights that I currently have by just adding more flats to the calibration pot.

Must also have a look at darks and bias

regards ray

Edit: Just redid the analysis using full SNR rather than just noise and the requirement is not so stringent - you need flat electrons > 10x sky electrons to get out of the region where flat noise is a problem. Am now reasonably happy with the analysis - now to do some tests to validate.

[RickS]

That appears to reflect my recent experience, Ray. I'm away from my scope and data for a few weeks but will do some experiments when I get back. I look forward to your results...

Cheers,
Rick.

[gregbradley]

There is a natural limitation of how many flats you can get if you take dusk/dawn flats. That is how long it takes to expose, download and save in rapidly changing light conditions.

This would also be an argument for using artificial light flats as there is no rush.

Greg.

[Paul Haese]

Hmmm ok, will give that a go. I usually use around 7 or so flats on the RC12. This has been largely a guess. So I will get a bit more precise about this and work out the numbers required. On the TSA I have been using 16 flats on each filter.

These days I tend to do flats on the single colour I am imaging that night.

[Shiraz]

thanks for the feedback and info.

Reworked the theory and will attempt to validate it tonight if all goes well - will start a new thread, since it could possibly be useful to others who might not read this one.

Looks like you can control flat-induced noise simply by making sure that you have at least 10x as many total flat electrons as you have total sky electrons. eg, if you take 100 lights where the sky background in each is 3,000ADU above the bias, you need to take 100 flats if you use 30,000 ADU flats. It has nothing to do with how many subs - just how many electrons - so if you took flats with say 10,000 ADU in the above example, you would need to take 300 of them. If you image for a long time, take a lot of flats, if you have bright sky, take a lot of flats.

Not yet sure how much of an issue colour is for noise reduction - guess that dust on filters might make separate flats worthwhile for getting rid of bunnies if that is a problem. Quite a bit of other interesting stuff to look at as well, including the possibility that past images can be improved by using two different sets of flats - one to control noise and the other to get rid of vignetting etc :).
Regards ray

[Paul Haese]

Holy hell. My subs are around 3300ADU so I need 33000 ADU flats. So if I take flats at 30000 ADU I will need 10 flats; is that right? What about if I take the at 26000? Am I to assume it would 13? Confused now. :shrug:

[Shiraz]

yes, no and no :thumbsup: you need only one 33,000ADU flat for each 3,300ADU light to get 10x. The only question is if you are including the bias in your 3300 for the lights - it will reduce the numbers a bit if you have not subtracted the bias. Regardless, a good enough rule of thumb for your system would be to take one 30,000 flat for every light - it is not critical that you get it exact but err on the "longer" side.

30,000ADU flats are good to get above read noise and below saturation - but you could use other flats if you compensate. For example if you wanted to use flats of 15,000 ADU, you would need twice as many of them as you would need of 30,000 ADU flats. the total ADU in all of your flats just has to add up to 10x the total sky ADU summed over your light frames.

making a few guesses on your imaging system characteristics, you would need maybe 30 flats of 30,000 ADU for each 10 hours of imaging.

[Paul Haese]

I will give it a go and get back to you with the results. It is easy to get all the flats I need for each project. Its worth a try to get a defined result.