Go Back   IceInSpace > Equipment > Astrophotography and Imaging Equipment and Discussions

Reply
 
Thread Tools Rate Thread
  #1  
Old 20-07-2015, 10:40 PM
Garbz (Chris)
Registered User

Garbz is offline
 
Join Date: May 2012
Location: Brisbane
Posts: 639
[SOVLED] Flats are overcorrecting.

Well I'm officially bashing my head against the wall. After many years of no issues I can't for the life of me make flats work. I just don't get it. For some reason I am now unable to put a photo together.

My flats are overcorrecting. The vignetting has gone the opposite way and the dark dust spots have turned to light dust spots and I'm at whits end as to what on earth is going on at this point.

In the attachment you can see the examples of my problem. On the left is an overcorrected stack where the flats were generated by pointing at the sky with a white t-shirt. In the centre was the original stack. This is pretty much normal that it looks so green. On the right was another attempt at stacking tonight again made with a tshirt but this time lit by the fluros in the house. The fluros make a very orange flat that results in a huge blue tint to the final image.

In the second picture are the photos with an unlinked screen transfer function.

In the third photo is a shot of one of the flats and some statistics of what I'm doing.

/Edit: I should also mention that relying on DBE along is not an option. My sensor is filthy and the dust spots will require the flat frames to correct.

So I was targeting the lower 1/3rd of the histogram and the flats average at around 15000 ADU. This low value was a second attempt as the first attempt which generated the left picture I thought I may have had had problems with non-linearity. But I can't resolve that right now.

The flats had bias and darks set so I loaded everything into the dark frame selector including a set of 300second darks for the light frames and a set of 30second darks for the flat frames.

I'm at wits-end. I have no idea what I'm doing wrong anymore. Has anyone got a clue?

Solution

My lights and flat frames were taken with a different gain and offset value from my darks and bias master frames which I had in my library. I blame a software update which may have reset the settings in EZCAP.

Anyway the wrong gain settings in the bias frames introduced a scaling issue when calibrating the flat frames. Flat + Lights themselves worked just fine. It was only when Bias frames were added that they failed to correct properly.

Thanks all for your help.
Attached Thumbnails
Click for full-size image (1.JPG)
118.3 KB42 views
Click for full-size image (2.JPG)
143.9 KB40 views
Click for full-size image (3.JPG)
79.5 KB39 views

Last edited by Garbz; 31-07-2015 at 12:23 PM.
Reply With Quote
  #2  
Old 20-07-2015, 10:54 PM
jase (Jason)
Registered User

jase is offline
 
Join Date: Sep 2006
Location: Melbourne, Victoria
Posts: 3,916
You may be able to use pixel math to correct them. I've done so in past when desperate. Not ideal. Better to reshoot.
Reply With Quote
  #3  
Old 20-07-2015, 11:50 PM
Garbz (Chris)
Registered User

Garbz is offline
 
Join Date: May 2012
Location: Brisbane
Posts: 639
Quote:
Originally Posted by jase View Post
You may be able to use pixel math to correct them. I've done so in past when desperate. Not ideal. Better to reshoot.
Was going to start down this path but I really want to resolve the root cause. In 3 years of this hobby I haven't had problems like this, so it must be something I'm doing wrong right now.
Reply With Quote
  #4  
Old 21-07-2015, 12:05 AM
rally
Registered User

rally is offline
 
Join Date: Sep 2007
Location: Australia
Posts: 896
Chris,

What programs and steps are you using for each step ?

I'm wondering if its an image file format problem.
Had a similar sort of thing happen before with flats developed in one software suite and then applied in another

Are these Master flats or single flats - if so what is the process for creating the masters ?

Rally
Reply With Quote
  #5  
Old 21-07-2015, 09:41 AM
troypiggo's Avatar
troypiggo (Troy)
Bust Duster

troypiggo is offline
 
Join Date: Nov 2008
Location: Brisbane, Australia
Posts: 4,814
Can you post the master flat stretched with auto STF? You're using a OSC camera? Are you using the BatchPreProcessing script in PI? Screenshots of each tab?
Reply With Quote
  #6  
Old 21-07-2015, 10:44 AM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,582
What happens if you calibrate a few flat frames with the master flat, Chris? If everything is working as it should that will give you an evenly illuminated result with no visible structures.

If you still get bad results then that points to a problem with capturing and/or creating the master flat. If it works, then that points to some sort of compatibility issue between flats and lights.

A bit of the old divide and conquer...

Cheers,
Rick.
Reply With Quote
  #7  
Old 21-07-2015, 10:59 AM
LewisM's Avatar
LewisM
Novichok test rabbit

LewisM is offline
 
Join Date: Aug 2012
Location: Somewhere in the cosmos...
Posts: 9,093
Seriously, if your sensor is so filthy, CLEAN IT! I really think flats are a lazy excuse sometimes for a clean sensor.

I keep my sensor as spotless as possible. Examining any of my fits you won't find any dust motes to speak of. I find I am better off manually removing any residual defects by zooming in and using a clone tool.

Some clean, some don't. Take your pick. Why not the best of both worlds - a clean sensor cover with any minor defects flat corrected?
Reply With Quote
  #8  
Old 21-07-2015, 11:06 AM
multiweb's Avatar
multiweb (Marc)
ze frogginator

multiweb is offline
 
Join Date: Oct 2007
Location: Hinchinbrook
Posts: 18,008
I don't know your routine for taking flats but if they are over correcting it is likely they are not scaled properly. To scale them you need to subtract a bias of equal length (sub time) at the exact same temperature you've shot your flat frames. If this still doesn't work you need to reduce or increase your flat times (and bias) until you hit the spot. Good luck.

Quote:
Originally Posted by LewisM View Post
Seriously, if your sensor is so filthy, CLEAN IT! I really think flats are a lazy excuse sometimes for a clean sensor.
Really? That's a bit simplistic Lewis. How do you deal with dust motes on filters, correctors, lenses then? Get a mop?

Last edited by multiweb; 21-07-2015 at 11:51 AM.
Reply With Quote
  #9  
Old 21-07-2015, 11:08 AM
LewisM's Avatar
LewisM
Novichok test rabbit

LewisM is offline
 
Join Date: Aug 2012
Location: Somewhere in the cosmos...
Posts: 9,093
all my filters are spotless too, and the objective is well maintained.

No mop, just proper technique for cleaning like NASA uses on it's mirrors..
Reply With Quote
  #10  
Old 21-07-2015, 11:12 AM
multiweb's Avatar
multiweb (Marc)
ze frogginator

multiweb is offline
 
Join Date: Oct 2007
Location: Hinchinbrook
Posts: 18,008
Quote:
Originally Posted by LewisM View Post
all my filters are spotless too, and the objective is well maintained.

No mop, just proper technique for cleaning like NASA uses on it's mirrors..
Good to hear you clean to NASA standards. You should elaborate your technique in the DIY section

Last edited by multiweb; 21-07-2015 at 11:51 AM.
Reply With Quote
  #11  
Old 21-07-2015, 02:50 PM
Garbz (Chris)
Registered User

Garbz is offline
 
Join Date: May 2012
Location: Brisbane
Posts: 639
Quote:
Originally Posted by rally View Post
Chris,

What programs and steps are you using for each step ?

I'm wondering if its an image file format problem.
Unlikely to be a file program. Taking the original FITS straight from the camera and through Pixinsight. I'm using Master Bias and Master Dark libraries like I always have, and I'm using individual flat files taken a few days after I took the light files but without any modification to the imaging train.

Quote:
Originally Posted by troypiggo View Post
Can you post the master flat stretched with auto STF? You're using a OSC camera? Are you using the BatchPreProcessing script in PI? Screenshots of each tab?
Screenshot of the master flat with an AutoSTF is attached. It's the one of the left. Yes OSC camera. All batch pre-processing. Most values are left at default except that I use overscan correction, master dark and master bias frames. I typically do integration outside of the script and use it to calibrate debeyer and register only, but it's not an integration problem the result is the same regardless of if I use the script or not.

Quote:
Originally Posted by RickS View Post
What happens if you calibrate a few flat frames with the master flat, Chris? If everything is working as it should that will give you an evenly illuminated result with no visible structures.
You just gave me an idea. I've attached an example with master on the left, original in the centre, and calibrated with the heck stretched out of it on the right. The result shows some problem to the left of the frame but no real issue with vignetting or dust spots. I've calibrated all flats and all showed the same kind of response as the right most image. This could point to a problem with either the darks or the bias frames. I'll do a bit more digging. But what is absent from this picture is problems of vignetting and dust motes which are an issue for the main image.

Quote:
Originally Posted by LewisM View Post
Seriously, if your sensor is so filthy, CLEAN IT! I really think flats are a lazy excuse sometimes for a clean sensor.
This is the least useful and most DANGEROUS thing you have ever posted. I'm trying to resolve a problem with existing image data and you're suggesting modifying the imaging train potentially destroying every chance I have of correcting that data. I'm happy that you have a perfect imaging system but your suggestion has ZERO to do with the problem at hand. Please stop helping.
Attached Thumbnails
Click for full-size image (1.jpg)
194.9 KB24 views
Reply With Quote
  #12  
Old 21-07-2015, 02:59 PM
multiweb's Avatar
multiweb (Marc)
ze frogginator

multiweb is offline
 
Join Date: Oct 2007
Location: Hinchinbrook
Posts: 18,008
Quote:
Originally Posted by RickS View Post
What happens if you calibrate a few flat frames with the master flat, Chris? If everything is working as it should that will give you an evenly illuminated result with no visible structures.
It will very likely correct the flat to an even illumination but unfortunately that does not guarantee that it will flat frame the light subs correctly. It is a scaling issue.
Reply With Quote
  #13  
Old 21-07-2015, 03:56 PM
troypiggo's Avatar
troypiggo (Troy)
Bust Duster

troypiggo is offline
 
Join Date: Nov 2008
Location: Brisbane, Australia
Posts: 4,814
Quote:
Originally Posted by LewisM View Post
Seriously, if your sensor is so filthy, CLEAN IT! I really think flats are a lazy excuse sometimes for a clean sensor.

I keep my sensor as spotless as possible. Examining any of my fits you won't find any dust motes to speak of. I find I am better off manually removing any residual defects by zooming in and using a clone tool.

Some clean, some don't. Take your pick. Why not the best of both worlds - a clean sensor cover with any minor defects flat corrected?
Flat calibration is more than just dust spots. For me the biggest benefit of flats is getting rid of vignetting. And it's infinitely quicker/easier/more reliable than trying to correct vignetting in post using ABE/DBE or something.
Reply With Quote
  #14  
Old 21-07-2015, 03:59 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,582
Quote:
Originally Posted by multiweb View Post
It will very likely correct the flat to an even illumination but unfortunately that does not guarantee that it will flat frame the light subs correctly. It is a scaling issue.
If the master flat is scaled wrongly then it will overcorrect an individual flat too, won't it Marc? The flat is no different to any other light apart from probably being brighter overall.

Cheers,
Rick.
Reply With Quote
  #15  
Old 21-07-2015, 04:05 PM
troypiggo's Avatar
troypiggo (Troy)
Bust Duster

troypiggo is offline
 
Join Date: Nov 2008
Location: Brisbane, Australia
Posts: 4,814
Quote:
Originally Posted by Garbz View Post
...
Screenshot of the master flat with an AutoSTF is attached. It's the one of the left. Yes OSC camera. All batch pre-processing. Most values are left at default except that I use overscan correction, master dark and master bias frames. I typically do integration outside of the script and use it to calibrate debeyer and register only, but it's not an integration problem the result is the same regardless of if I use the script or not.
...
You just gave me an idea. I've attached an example with master on the left, original in the centre, and calibrated with the heck stretched out of it on the right. The result shows some problem to the left of the frame but no real issue with vignetting or dust spots. I've calibrated all flats and all showed the same kind of response as the right most image. This could point to a problem with either the darks or the bias frames. I'll do a bit more digging. But what is absent from this picture is problems of vignetting and dust motes which are an issue for the main image.
...
I don't know of many people that use overscan other than Rick. I don't know if or how he does flat calibration, but if he does, then he clearly isn't having this same issue or he'd have mentioned it.

I did have a play some time ago with overscan, and I have in the past had a similar issue with flats not correcting properly. It was so long ago that I don't recall if the flats issue was happening while I was testing overscan. I've since stopped using overscan as it wasn't benefiting my images appreciably.

What I'm getting at - perhaps try some bias, dark, light, flat files without the overscan to see if you still have the issue there? Just to rule it out.
Reply With Quote
  #16  
Old 21-07-2015, 04:05 PM
multiweb's Avatar
multiweb (Marc)
ze frogginator

multiweb is offline
 
Join Date: Oct 2007
Location: Hinchinbrook
Posts: 18,008
Quote:
Originally Posted by RickS View Post
If the master flat is scaled wrongly then it will overcorrect an individual flat too, won't it Marc? The flat is no different to any other light apart from probably being brighter overall.

Cheers,
Rick.
You'd think so. I checked my flats this way when I collect them. Make a master of let's say 10 subs then 20, etc... then flat field a sub to check. They always flat field correctly. If I have the wrong bias which I like to call "dark flat" as opposed to "light flat" then the scaling is all over the place. I never successfully managed to pixel math them to get a proper flat fielding either. As everybody else probably did I've also done a lot of reading in the past trying to figure it out. I know the "maths" say 1/3 or full well, blahblahblah. In practice it never worked for me. It was trial and error until I got the levels I wanted. I don't try to understand why anymore. I just use the exposure times and ADU that work for my camera and scopes. I've found it to be scope dependant and also filter dependant unfortunately. I always flatfield and fully calibrate all my light subs. Always.
Reply With Quote
  #17  
Old 21-07-2015, 04:23 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,582
Quote:
Originally Posted by troypiggo View Post
I don't know of many people that use overscan other than Rick. I don't know if or how he does flat calibration, but if he does, then he clearly isn't having this same issue or he'd have mentioned it.
Yep, I use overscan and do traditional bias/dark/flat calibration. I dump everything into BPP and it just works.

Quote:
Originally Posted by multiweb View Post
You'd think so. I checked my flats this way when I collect them. Make a master of let's say 10 subs then 20, etc... then flat field a sub to check. They always flat field correctly. If I have the wrong bias which I like to call "dark flat" as opposed to "light flat" then the scaling is all over the place. I never successfully managed to pixel math them to get a proper flat fielding either. As everybody else probably did I've also done a lot of reading in the past trying to figure it out. I know the "maths" say 1/3 or full well, blahblahblah. In practice it never worked for me. It was trial and error until I got the levels I wanted. I don't try to understand why anymore. I just use the exposure times and ADU that work for my camera and scopes. I've found it to be scope dependant and also filter dependant unfortunately. I always flatfield and fully calibrate all my light subs. Always.
Flat framing does seem to be a black art. I have been fortunate and it has always just worked for me. If it hadn't I guess I might have developed some better diagnostic tools and would have more useful suggestions for Chris

That said, apparently we've got some problem with dawn and dusk flats behaving differently at SRO. I haven't had a chance to dig into that yet but maybe my first flat framing nightmare is about to start

Cheers,
Rick.
Reply With Quote
  #18  
Old 21-07-2015, 05:02 PM
Garbz (Chris)
Registered User

Garbz is offline
 
Join Date: May 2012
Location: Brisbane
Posts: 639
Quote:
Originally Posted by troypiggo View Post
I don't know of many people that use overscan other than Rick.
The reason I started doing overscan correction is that during the calibration step BP will auto-crop the image area out and my camera gives me the full overscanned frame. I started because one particularly dark object in a dark part of the sky wasn't registering. Rick made an off the cuff suggestion about turning on overscan, not to fix the issue I was having but in general and that's when I realised the problem. The start detection picking up thousands of stars; actually it was picking up noise in the overscan area because of how stretched the final images were. I haven't turned it off since.


Quote:
Originally Posted by RickS View Post
Flat framing does seem to be a black art. I have been fortunate and it has always just worked for me. If it hadn't I guess I might have developed some better diagnostic tools and would have more useful suggestions for Chris
Same here. I don't understand why I can't figure this out all of a sudden, 3 years into this hobby ....

Quote:
Originally Posted by multiweb View Post
If I have the wrong bias which I like to call "dark flat" as opposed to "light flat" then the scaling is all over the place.
I may just try an experiment with only flats and lights and see what I end up with. Then I'll add bias. Maybe I've got a problem with either my bias or darks, but they shouldn't have changed either.
Reply With Quote
  #19  
Old 21-07-2015, 06:31 PM
Garbz (Chris)
Registered User

Garbz is offline
 
Join Date: May 2012
Location: Brisbane
Posts: 639
Yay partial result.

The field is flat, but now I need to figure out what I forgot to do. I can't remember if it was bias or darks, but the noise profile is absolutely horrendous.

I did this entirely manually using master frames and then stepping through calibration, debeyer and registration bit by bit.
Attached Thumbnails
Click for full-size image (1.JPG)
101.9 KB39 views
Reply With Quote
  #20  
Old 21-07-2015, 06:50 PM
RickS's Avatar
RickS (Rick)
PI cult recruiter

RickS is offline
 
Join Date: Apr 2010
Location: Brisbane
Posts: 10,582
I'm sure you can do it again if you've done it once, Chris!
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 03:45 PM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Advertisement
OzScopes Authorised Dealer
Advertisement
SkyWatcher Australia
Advertisement
Celestron Australia
Advertisement
Bintel
Advertisement
Meade Australia
Advertisement
Lunatico Astronomical
Advertisement
NexDome Observatories
Advertisement
Astronomy and Electronics Centre
Advertisement