Well I'm officially bashing my head against the wall. After many years of no issues I can't for the life of me make flats work. I just don't get it. For some reason I am now unable to put a photo together.
My flats are overcorrecting. The vignetting has gone the opposite way and the dark dust spots have turned to light dust spots and I'm at whits end as to what on earth is going on at this point.
In the attachment you can see the examples of my problem. On the left is an overcorrected stack where the flats were generated by pointing at the sky with a white t-shirt. In the centre was the original stack. This is pretty much normal that it looks so green. On the right was another attempt at stacking tonight again made with a tshirt but this time lit by the fluros in the house. The fluros make a very orange flat that results in a huge blue tint to the final image.
In the second picture are the photos with an unlinked screen transfer function.
In the third photo is a shot of one of the flats and some statistics of what I'm doing.
/Edit: I should also mention that relying on DBE along is not an option. My sensor is filthy and the dust spots will require the flat frames to correct.
So I was targeting the lower 1/3rd of the histogram and the flats average at around 15000 ADU. This low value was a second attempt as the first attempt which generated the left picture I thought I may have had had problems with non-linearity. But I can't resolve that right now.
The flats had bias and darks set so I loaded everything into the dark frame selector including a set of 300second darks for the light frames and a set of 30second darks for the flat frames.
I'm at wits-end. I have no idea what I'm doing wrong anymore. Has anyone got a clue?
Solution
My lights and flat frames were taken with a different gain and offset value from my darks and bias master frames which I had in my library. I blame a software update which may have reset the settings in EZCAP.
Anyway the wrong gain settings in the bias frames introduced a scaling issue when calibrating the flat frames. Flat + Lights themselves worked just fine. It was only when Bias frames were added that they failed to correct properly.
You may be able to use pixel math to correct them. I've done so in past when desperate. Not ideal. Better to reshoot.
Was going to start down this path but I really want to resolve the root cause. In 3 years of this hobby I haven't had problems like this, so it must be something I'm doing wrong right now.
What programs and steps are you using for each step ?
I'm wondering if its an image file format problem.
Had a similar sort of thing happen before with flats developed in one software suite and then applied in another
Are these Master flats or single flats - if so what is the process for creating the masters ?
Can you post the master flat stretched with auto STF? You're using a OSC camera? Are you using the BatchPreProcessing script in PI? Screenshots of each tab?
What happens if you calibrate a few flat frames with the master flat, Chris? If everything is working as it should that will give you an evenly illuminated result with no visible structures.
If you still get bad results then that points to a problem with capturing and/or creating the master flat. If it works, then that points to some sort of compatibility issue between flats and lights.
Seriously, if your sensor is so filthy, CLEAN IT! I really think flats are a lazy excuse sometimes for a clean sensor.
I keep my sensor as spotless as possible. Examining any of my fits you won't find any dust motes to speak of. I find I am better off manually removing any residual defects by zooming in and using a clone tool.
Some clean, some don't. Take your pick. Why not the best of both worlds - a clean sensor cover with any minor defects flat corrected?
I don't know your routine for taking flats but if they are over correcting it is likely they are not scaled properly. To scale them you need to subtract a bias of equal length (sub time) at the exact same temperature you've shot your flat frames. If this still doesn't work you need to reduce or increase your flat times (and bias) until you hit the spot. Good luck.
Quote:
Originally Posted by LewisM
Seriously, if your sensor is so filthy, CLEAN IT! I really think flats are a lazy excuse sometimes for a clean sensor.
Really? That's a bit simplistic Lewis. How do you deal with dust motes on filters, correctors, lenses then? Get a mop?
What programs and steps are you using for each step ?
I'm wondering if its an image file format problem.
Unlikely to be a file program. Taking the original FITS straight from the camera and through Pixinsight. I'm using Master Bias and Master Dark libraries like I always have, and I'm using individual flat files taken a few days after I took the light files but without any modification to the imaging train.
Quote:
Originally Posted by troypiggo
Can you post the master flat stretched with auto STF? You're using a OSC camera? Are you using the BatchPreProcessing script in PI? Screenshots of each tab?
Screenshot of the master flat with an AutoSTF is attached. It's the one of the left. Yes OSC camera. All batch pre-processing. Most values are left at default except that I use overscan correction, master dark and master bias frames. I typically do integration outside of the script and use it to calibrate debeyer and register only, but it's not an integration problem the result is the same regardless of if I use the script or not.
Quote:
Originally Posted by RickS
What happens if you calibrate a few flat frames with the master flat, Chris? If everything is working as it should that will give you an evenly illuminated result with no visible structures.
You just gave me an idea. I've attached an example with master on the left, original in the centre, and calibrated with the heck stretched out of it on the right. The result shows some problem to the left of the frame but no real issue with vignetting or dust spots. I've calibrated all flats and all showed the same kind of response as the right most image. This could point to a problem with either the darks or the bias frames. I'll do a bit more digging. But what is absent from this picture is problems of vignetting and dust motes which are an issue for the main image.
Quote:
Originally Posted by LewisM
Seriously, if your sensor is so filthy, CLEAN IT! I really think flats are a lazy excuse sometimes for a clean sensor.
This is the least useful and most DANGEROUS thing you have ever posted. I'm trying to resolve a problem with existing image data and you're suggesting modifying the imaging train potentially destroying every chance I have of correcting that data. I'm happy that you have a perfect imaging system but your suggestion has ZERO to do with the problem at hand. Please stop helping.
What happens if you calibrate a few flat frames with the master flat, Chris? If everything is working as it should that will give you an evenly illuminated result with no visible structures.
It will very likely correct the flat to an even illumination but unfortunately that does not guarantee that it will flat frame the light subs correctly. It is a scaling issue.
Seriously, if your sensor is so filthy, CLEAN IT! I really think flats are a lazy excuse sometimes for a clean sensor.
I keep my sensor as spotless as possible. Examining any of my fits you won't find any dust motes to speak of. I find I am better off manually removing any residual defects by zooming in and using a clone tool.
Some clean, some don't. Take your pick. Why not the best of both worlds - a clean sensor cover with any minor defects flat corrected?
Flat calibration is more than just dust spots. For me the biggest benefit of flats is getting rid of vignetting. And it's infinitely quicker/easier/more reliable than trying to correct vignetting in post using ABE/DBE or something.
It will very likely correct the flat to an even illumination but unfortunately that does not guarantee that it will flat frame the light subs correctly. It is a scaling issue.
If the master flat is scaled wrongly then it will overcorrect an individual flat too, won't it Marc? The flat is no different to any other light apart from probably being brighter overall.
...
Screenshot of the master flat with an AutoSTF is attached. It's the one of the left. Yes OSC camera. All batch pre-processing. Most values are left at default except that I use overscan correction, master dark and master bias frames. I typically do integration outside of the script and use it to calibrate debeyer and register only, but it's not an integration problem the result is the same regardless of if I use the script or not.
...
You just gave me an idea. I've attached an example with master on the left, original in the centre, and calibrated with the heck stretched out of it on the right. The result shows some problem to the left of the frame but no real issue with vignetting or dust spots. I've calibrated all flats and all showed the same kind of response as the right most image. This could point to a problem with either the darks or the bias frames. I'll do a bit more digging. But what is absent from this picture is problems of vignetting and dust motes which are an issue for the main image.
...
I don't know of many people that use overscan other than Rick. I don't know if or how he does flat calibration, but if he does, then he clearly isn't having this same issue or he'd have mentioned it.
I did have a play some time ago with overscan, and I have in the past had a similar issue with flats not correcting properly. It was so long ago that I don't recall if the flats issue was happening while I was testing overscan. I've since stopped using overscan as it wasn't benefiting my images appreciably.
What I'm getting at - perhaps try some bias, dark, light, flat files without the overscan to see if you still have the issue there? Just to rule it out.
If the master flat is scaled wrongly then it will overcorrect an individual flat too, won't it Marc? The flat is no different to any other light apart from probably being brighter overall.
Cheers,
Rick.
You'd think so. I checked my flats this way when I collect them. Make a master of let's say 10 subs then 20, etc... then flat field a sub to check. They always flat field correctly. If I have the wrong bias which I like to call "dark flat" as opposed to "light flat" then the scaling is all over the place. I never successfully managed to pixel math them to get a proper flat fielding either. As everybody else probably did I've also done a lot of reading in the past trying to figure it out. I know the "maths" say 1/3 or full well, blahblahblah. In practice it never worked for me. It was trial and error until I got the levels I wanted. I don't try to understand why anymore. I just use the exposure times and ADU that work for my camera and scopes. I've found it to be scope dependant and also filter dependant unfortunately. I always flatfield and fully calibrate all my light subs. Always.
I don't know of many people that use overscan other than Rick. I don't know if or how he does flat calibration, but if he does, then he clearly isn't having this same issue or he'd have mentioned it.
Yep, I use overscan and do traditional bias/dark/flat calibration. I dump everything into BPP and it just works.
Quote:
Originally Posted by multiweb
You'd think so. I checked my flats this way when I collect them. Make a master of let's say 10 subs then 20, etc... then flat field a sub to check. They always flat field correctly. If I have the wrong bias which I like to call "dark flat" as opposed to "light flat" then the scaling is all over the place. I never successfully managed to pixel math them to get a proper flat fielding either. As everybody else probably did I've also done a lot of reading in the past trying to figure it out. I know the "maths" say 1/3 or full well, blahblahblah. In practice it never worked for me. It was trial and error until I got the levels I wanted. I don't try to understand why anymore. I just use the exposure times and ADU that work for my camera and scopes. I've found it to be scope dependant and also filter dependant unfortunately. I always flatfield and fully calibrate all my light subs. Always.
Flat framing does seem to be a black art. I have been fortunate and it has always just worked for me. If it hadn't I guess I might have developed some better diagnostic tools and would have more useful suggestions for Chris
That said, apparently we've got some problem with dawn and dusk flats behaving differently at SRO. I haven't had a chance to dig into that yet but maybe my first flat framing nightmare is about to start
I don't know of many people that use overscan other than Rick.
The reason I started doing overscan correction is that during the calibration step BP will auto-crop the image area out and my camera gives me the full overscanned frame. I started because one particularly dark object in a dark part of the sky wasn't registering. Rick made an off the cuff suggestion about turning on overscan, not to fix the issue I was having but in general and that's when I realised the problem. The start detection picking up thousands of stars; actually it was picking up noise in the overscan area because of how stretched the final images were. I haven't turned it off since.
Quote:
Originally Posted by RickS
Flat framing does seem to be a black art. I have been fortunate and it has always just worked for me. If it hadn't I guess I might have developed some better diagnostic tools and would have more useful suggestions for Chris
Same here. I don't understand why I can't figure this out all of a sudden, 3 years into this hobby ....
Quote:
Originally Posted by multiweb
If I have the wrong bias which I like to call "dark flat" as opposed to "light flat" then the scaling is all over the place.
I may just try an experiment with only flats and lights and see what I end up with. Then I'll add bias. Maybe I've got a problem with either my bias or darks, but they shouldn't have changed either.
The field is flat, but now I need to figure out what I forgot to do. I can't remember if it was bias or darks, but the noise profile is absolutely horrendous.
I did this entirely manually using master frames and then stepping through calibration, debeyer and registration bit by bit.