[SOVLED] Flats are overcorrecting.

[Garbz]

Well I'm officially bashing my head against the wall. After many years of no issues I can't for the life of me make flats work. I just don't get it. For some reason I am now unable to put a photo together.

My flats are overcorrecting. The vignetting has gone the opposite way and the dark dust spots have turned to light dust spots and I'm at whits end as to what on earth is going on at this point.

In the attachment you can see the examples of my problem. On the left is an overcorrected stack where the flats were generated by pointing at the sky with a white t-shirt. In the centre was the original stack. This is pretty much normal that it looks so green. On the right was another attempt at stacking tonight again made with a tshirt but this time lit by the fluros in the house. The fluros make a very orange flat that results in a huge blue tint to the final image.

In the second picture are the photos with an unlinked screen transfer function.

In the third photo is a shot of one of the flats and some statistics of what I'm doing.

/Edit: I should also mention that relying on DBE along is not an option. My sensor is filthy and the dust spots will require the flat frames to correct.

So I was targeting the lower 1/3rd of the histogram and the flats average at around 15000 ADU. This low value was a second attempt as the first attempt which generated the left picture I thought I may have had had problems with non-linearity. But I can't resolve that right now.

The flats had bias and darks set so I loaded everything into the dark frame selector including a set of 300second darks for the light frames and a set of 30second darks for the flat frames.

I'm at wits-end. I have no idea what I'm doing wrong anymore. Has anyone got a clue?

Solution

My lights and flat frames were taken with a different gain and offset value from my darks and bias master frames which I had in my library. I blame a software update which may have reset the settings in EZCAP.

Anyway the wrong gain settings in the bias frames introduced a scaling issue when calibrating the flat frames. Flat + Lights themselves worked just fine. It was only when Bias frames were added that they failed to correct properly.

Thanks all for your help.

[jase]

You may be able to use pixel math to correct them. I've done so in past when desperate. Not ideal. Better to reshoot.

[Garbz]

Was going to start down this path but I really want to resolve the root cause. In 3 years of this hobby I haven't had problems like this, so it must be something I'm doing wrong right now.

[rally]

Chris,

What programs and steps are you using for each step ?

I'm wondering if its an image file format problem.
Had a similar sort of thing happen before with flats developed in one software suite and then applied in another

Are these Master flats or single flats - if so what is the process for creating the masters ?

Rally

[troypiggo]

Can you post the master flat stretched with auto STF? You're using a OSC camera? Are you using the BatchPreProcessing script in PI? Screenshots of each tab?

[RickS]

What happens if you calibrate a few flat frames with the master flat, Chris? If everything is working as it should that will give you an evenly illuminated result with no visible structures.

If you still get bad results then that points to a problem with capturing and/or creating the master flat. If it works, then that points to some sort of compatibility issue between flats and lights.

A bit of the old divide and conquer...

Cheers,
Rick.

[LewisM]

Seriously, if your sensor is so filthy, CLEAN IT! I really think flats are a lazy excuse sometimes for a clean sensor.

I keep my sensor as spotless as possible. Examining any of my fits you won't find any dust motes to speak of. I find I am better off manually removing any residual defects by zooming in and using a clone tool.

Some clean, some don't. Take your pick. Why not the best of both worlds - a clean sensor cover with any minor defects flat corrected?

[multiweb]

I don't know your routine for taking flats but if they are over correcting it is likely they are not scaled properly. To scale them you need to subtract a bias of equal length (sub time) at the exact same temperature you've shot your flat frames. If this still doesn't work you need to reduce or increase your flat times (and bias) until you hit the spot. Good luck. :thumbsup:



Really? :rolleyes: That's a bit simplistic Lewis. How do you deal with dust motes on filters, correctors, lenses then? Get a mop? :lol:

[LewisM]

all my filters are spotless too, and the objective is well maintained.

No mop, just proper technique for cleaning like NASA uses on it's mirrors.. :)

[multiweb]

Good to hear you clean to NASA standards. You should elaborate your technique in the DIY section :thumbsup:

[Garbz]

Unlikely to be a file program. Taking the original FITS straight from the camera and through Pixinsight. I'm using Master Bias and Master Dark libraries like I always have, and I'm using individual flat files taken a few days after I took the light files but without any modification to the imaging train.



Screenshot of the master flat with an AutoSTF is attached. It's the one of the left. Yes OSC camera. All batch pre-processing. Most values are left at default except that I use overscan correction, master dark and master bias frames. I typically do integration outside of the script and use it to calibrate debeyer and register only, but it's not an integration problem the result is the same regardless of if I use the script or not.



You just gave me an idea. I've attached an example with master on the left, original in the centre, and calibrated with the heck stretched out of it on the right. The result shows some problem to the left of the frame but no real issue with vignetting or dust spots. I've calibrated all flats and all showed the same kind of response as the right most image. This could point to a problem with either the darks or the bias frames. I'll do a bit more digging. But what is absent from this picture is problems of vignetting and dust motes which are an issue for the main image.



This is the least useful and most DANGEROUS thing you have ever posted. I'm trying to resolve a problem with existing image data and you're suggesting modifying the imaging train potentially destroying every chance I have of correcting that data. I'm happy that you have a perfect imaging system but your suggestion has ZERO to do with the problem at hand. Please stop helping.

[multiweb]

It will very likely correct the flat to an even illumination but unfortunately that does not guarantee that it will flat frame the light subs correctly. It is a scaling issue.

[troypiggo]

Flat calibration is more than just dust spots. For me the biggest benefit of flats is getting rid of vignetting. And it's infinitely quicker/easier/more reliable than trying to correct vignetting in post using ABE/DBE or something.

[RickS]

If the master flat is scaled wrongly then it will overcorrect an individual flat too, won't it Marc? The flat is no different to any other light apart from probably being brighter overall.

Cheers,
Rick.

[troypiggo]

I don't know of many people that use overscan other than Rick. I don't know if or how he does flat calibration, but if he does, then he clearly isn't having this same issue or he'd have mentioned it.

I did have a play some time ago with overscan, and I have in the past had a similar issue with flats not correcting properly. It was so long ago that I don't recall if the flats issue was happening while I was testing overscan. I've since stopped using overscan as it wasn't benefiting my images appreciably.

What I'm getting at - perhaps try some bias, dark, light, flat files without the overscan to see if you still have the issue there? Just to rule it out.

[multiweb]

You'd think so. I checked my flats this way when I collect them. Make a master of let's say 10 subs then 20, etc... then flat field a sub to check. They always flat field correctly. If I have the wrong bias which I like to call "dark flat" as opposed to "light flat" then the scaling is all over the place. I never successfully managed to pixel math them to get a proper flat fielding either. As everybody else probably did I've also done a lot of reading in the past trying to figure it out. I know the "maths" say 1/3 or full well, blahblahblah. In practice it never worked for me. It was trial and error until I got the levels I wanted. I don't try to understand why anymore. I just use the exposure times and ADU that work for my camera and scopes. I've found it to be scope dependant and also filter dependant unfortunately. I always flatfield and fully calibrate all my light subs. Always.

[RickS]

Yep, I use overscan and do traditional bias/dark/flat calibration. I dump everything into BPP and it just works.



Flat framing does seem to be a black art. I have been fortunate and it has always just worked for me. If it hadn't I guess I might have developed some better diagnostic tools and would have more useful suggestions for Chris :)

That said, apparently we've got some problem with dawn and dusk flats behaving differently at SRO. I haven't had a chance to dig into that yet but maybe my first flat framing nightmare is about to start :eyepop:

Cheers,
Rick.

[Garbz]

The reason I started doing overscan correction is that during the calibration step BP will auto-crop the image area out and my camera gives me the full overscanned frame. I started because one particularly dark object in a dark part of the sky wasn't registering. Rick made an off the cuff suggestion about turning on overscan, not to fix the issue I was having but in general and that's when I realised the problem. The start detection picking up thousands of stars; actually it was picking up noise in the overscan area because of how stretched the final images were. I haven't turned it off since.




Same here. I don't understand why I can't figure this out all of a sudden, 3 years into this hobby ....



I may just try an experiment with only flats and lights and see what I end up with. Then I'll add bias. Maybe I've got a problem with either my bias or darks, but they shouldn't have changed either.

[Garbz]

Yay partial result.

The field is flat, but now I need to figure out what I forgot to do. I can't remember if it was bias or darks, but the noise profile is absolutely horrendous.

I did this entirely manually using master frames and then stepping through calibration, debeyer and registration bit by bit.

[RickS]

I'm sure you can do it again if you've done it once, Chris!

[Garbz]

But can I do it again without the funny noise and vertical lines in the image this time. :-)

[gregbradley]

I have had problems with flats in the past. So yes it can suddenly go haywire when some simple step has been omitted or a few dodgy individual flats got into the master flat. I was playing around with some flats just last night to do with another thread and simple minor changes can make it go weirdly wrong.

Best to list exactly what steps you are taking and with what camera and scope, software and how you take your flats - dusk with a white cloth over the end of the scope, illuminated panel etc.

Its impossible to comment without knowing what your current procedure is.

Here at least is my procedure:

1. I take 6 to 12 flats at dusk or sometimes during the day with a white cloth cover over the end of my scope. In my case my observatory which is painted black on the inside walls lets in a small amount of light to make taking a flat possible. These work well. Otherwise I take dusk flats. It can be a race to take them and you need a camera with fairly fast download speed as you can see the ADU falling between subs. I increase the exposure length to compensate between filters.

2. I take bias perhaps 12 to 16 at the same temp.
3. Darks are about 16 subs and at the same temp and binning. I use sigma reject combine for the master dark using CCDstack. I do the same for bias.
4. Flats on a camera with a physical leaf type shutter like Apogee, FLI need a minimum of about 4 seconds exposure otherwise you see the shutter in the flat.
5. When applying to the same dark to the lights I do not use scaling but I tend to leave scaling on in CCDstack and bias subtract on.
6. I do median combine for flats and when applying the master flat to the images I check the subtract bias from the master flat. You can subtract the bias from the subexposure flats when making a master flat. I don't know it makes too much difference but for some reason I got better results doing it as I mentioned.
7. I check to see my luminance is flat fielded well with a trial and go hunting for what is different if I don't get a good result. I often do this while the camera is imaging so I've got something to do whilst I am waiting.

With a Sony sensor then you really don't need to do any of the above if your sensor and filters are clean which I prefer (of course before you start imaging). A bias subtract may be all that is needed unless you have dust donuts. So the above is the procedure for a Kodak sensor which are noisier and need the above to clean up. Or for a setup that has significant vignetting or dust donuts.

Is this similar to what you are doing? Do you scale your darks? Are you flats at a consistent ADU level more or less? I go for around 20-30K ADU. These work well. I have had some flats that seem too bright and get reverse to vignetting. I have had bad calibration when the darks do not match the lights precisely on temp and duration on some touchy scopes. Most though are fairly tolerant. Are you using 1x1 binned flats on 2x2 subexposures? I believe you can do that with CCDstack. I am unsure how well it works but perhaps.

I have at times used one flat from red for all RGB if I didn't get good flats for the others and the other coloured filters are clean with no dust donuts. Not ideal as perhaps they have dust donuts but if everything is clean I've gotten away with it. I usually have a separate master flat for each filter imaged with in exactly the same orientation as the filter wheel (perhaps not so important to match the orientation of the scope just the filter wheel and of course nothing changed in the optical train as well as the scope is focused and not way out of focus which could shift things.

The first thing I would do is check the master flat and make sure it looks like a usual flat that has worked for you over the years.

Greg.

[Atmos]

I personally consider flats to be for a lot more than just removing dust donuts. Unlike most on here though, I am also uninterested in doing astrometry which requires a good flat field above all else.

For the "perfect" flat field it must be taken at the same focus as what you are imaging at because light fall off from the centre to the edges changes as the focus does. On a purely cosmetic level, the dust donuts change with focus as well.

As for configuring ADU levels, this is quite easily calculated on camera. Most cameras have a linear response for 75-90% of saturation, this is tested by gradually increasing exposure time (the same way you would normally do flat) and plotting exposure time against average ADU. Typically you want to do 50-75% of saturation for a flat field which equates to 24,000-40,000 but this depends on the camera. As Greg mentioned, have them last at least 5 seconds if they have a shutter.

[Garbz]

I'm experimenting at the moment. I think I may have found the root cause to be settings taken at the time. I'm redoing darks right now and already have a new set of flat and biases. So stay tuned....

I would agree with flats being about more than just dust. Watch this space and hopefully I can fist pump the air tomorrow.

[RB]

Careful, that'll move the dust motes around.

:P

[h0ughy]

:rofl: i am following this thread carefully:thanx:

[Garbz]

Once the flats correct, I won't care anymore :-)

[Garbz]

AND THE WINNER IS:
Marc and Rick.

The problem turned out to be a screw up in image capturing. I blame upgrading my software but my Gain and Offset values for the camera were set at 14 and 133.

My dark library and bias frames had a gain and offset of 17 and 123 so the image calibration process was visually assaulting my frames quite badly. The crap bias frames would have introduced scaling issues.
I redid flats light night and then ran off 50 darks and 50 bias frames, and the result is below. Not sure where the gradient came from, but I don't care. DBE makes short work of that.

Funny story:

Actually I found the problem 2 days ago but when I ran the 50 darks and 50 bias frames I realised that I forgot to change the gain and offset values. I remembered 3 hours later, stopped the process, deleted the files and then noticed my telescope wasn't parked and was past the merridian, so I moved it parked it and then started taking darks and bias frames again.

Notice what I missed? 45minutes later I realised I still hadn't changed the gain and offset values. :doh:. So repeated the process set the gain and offset and then got 1 frame out before I decided it was bed time. There's really no helping me anymore is there :lol:


Anyway thanks all.

[gregbradley]

Nice find. Sometimes you just have lousy nights eh? Is this a QHY camera?

Greg.

[RickS]

Glad you got it resolved, Chris :)

[Garbz]

Yeah QHY10 why?

[gregbradley]

Other brands I have used don't allow you to set gain and offset as far as I know.They are set by the manufacturer. I wonder what the advantage is for allowing it? It seems like an unnecessary variable and no doubt one setting puts the camera at optimum which would be something worked out by the engineers and tests.

Greg.

[Atmos]

All you really need is a light box and 30-60 minutes so it can be done on a rainy day. The only reason I can think of for doing this is to allow the user to do the same thing you can do on a DSLR, increase the ISO if wanted.

[Garbz]

Or just not need to factory calibrate each unit :-)

[gregbradley]

I suppose you'd want to be careful about increasing the gain very much because I would expect you could lose dynamic range quickly or increase noise just like higher ISO's do.

ISO is a bit misleading in that you've got to capture the signal before you can amplify it. As I recall from school, 10 times zero is still zero!

Greg.

[Atmos]

Have to be very careful, not something that I would personally recommend doing!

[Garbz]

And yet there's an application that I use it for on a daily basis. Not everything requires the max dynamic range, specifically:

-Previewing: 2x binning + gain of around 40 allows me in 10 seconds to see if I have a faint outline of a nebula and I'm pointed in the right direction at the expense of horrendously clipped stars which are bleeding and drawing lines all over my picture.

-Focusing: doing a partial sensor download with the gain at almost as high as it will go makes short work of Bahtinov masks when focusing, even on a 1 second exposure.

Also lowest gain is not necessarily the highest dynamic range point on a sensor and binning has a big effect on gain and dynamic range so to maximise your dynamic range you want to run at a different gain when binning.

[Atmos]

I do totally agree with you, same thing I do on my DSLR. ISO up at Hi2 (~26,000) for focusing. It does have its uses, I was more referring to using it in capturing lights. You make a good point though, especially with the binning which is something that does need to be considered with cameras that allow you to set the values.

[gregbradley]

An interesting use. I can see that for focusing etc. I usually use 3x3 or 2x2 binning for focusing and framing. 1x1 is too slow and too large an image. For critical focus 1x1 is good but slow.

Not 100% sure if it this applies to CCDs but I don't see why not. Dynamic Range on DSLRs is highest at lowest native ISO. Dynamic range is defined by well depth divided by read noise. As far as I am aware gain is applied during read out but before analogue to digital conversion so in effect it would be added to the read noise. That's why on DSLRs they have a native ISO range (amplifier gain before digital conversion).

So you lower gain when binning to prevent burn out bright areas?

Greg.


Also lowest gain is not necessarily the highest dynamic range point on a sensor and binning has a big effect on gain and dynamic range so to maximise your dynamic range you want to run at a different gain when binning.[/QUOTE]