Image Processing Across Multiple Nights

[Atmos (Colin)]

This is probably going to sound like a bit of a noob question but I was wondering how people typically go about processing data taken over multiple evenings?

The question mainly goes towards image stacking. Is it better to...
1) Stack each individual night and the combine those "masters" together.
2) Combine the entire set of nights at once.

I have in mind Rolf's 150 hour marathons! Whether it is practical combining 40 odd images (masters) or hundreds.

[RickS (Rick)]

Stacking all the individual subs at once will provide maximum information for the rejection algorithm to chew on and should produce the best results. Splitting them up into large batches is going to be nearly as good but I definitely wouldn't do smaller batches, like one each night.

I have been able to stack hundreds of subs at one time with PI on a Windows system with a relatively modest configuration, at least compared to the average gaming computer On a Mac I think you're limited to a couple of hundred max because of a dumb OS limit on the maximum number of open file descriptors.

Cheers,
Rick.

[SkyViking (Rolf)]

Hi Colin, I've had to stack in batches in the past occasionally, for example for my 120 hour Centaurus A image, but that was merely because Maxim ran out of memory with too many images in the stack (1000+). After I switched to PixInsight I've never had an issue and would always recommend stacking the whole set together as Rick described above.

[Slawomir (Suavi)]

I second what Rick recommends in regards to stacking the lot at once vs small batches. Just need to be mindful of applying flats correctly to the subs from long marathons.

P.S. My 5-old all-in-one computer with 2GB RAM managed to stack 270 bias frames, each about 18 MB with PixInsight, but I needed to be very patient

[Atmos (Colin)]

I did 200 bias frames this afternoon and that only took a couple of minutes to stack, I've got a MacBook Pro and it seems to do the job well enough.

Sounds like keeping all of the individual image files seems like the way to go. Going to have to buy an external HD, my 250GB SSD isn't going to be enough for Astro purposes!!

[sil (Steve)]

Originally Posted by Atmos

Sounds like keeping all of the individual image files seems like the way to go.

Yep. I keep ALL source files I ever capture so I can go back and reprocess in the future or whatever. Not just for astrophotography but anything my cameras capture or raw scans of film. As you learn and software improves later on you'll be able to make better use of what you capture today. I've recently been throwing old AVI files into current versions of software I use and pulled out far better images than I was able to first time around. If you only keep your worked-over "final" image you limit improvement chances. With intact sources image you always have the best starting point.

[gregbradley]

Good thread. I was meaning to post the same question myself to see what others do.

I use CCDstack and you definitely cannot stack 100 images at once. You get memory errors. So I try to stack as many as CCDstack will allow which for Trius 694 11mb files is about 20 or 30 but for 16803 files its more like 15 perhaps 20.

This may be another reason to use PI then if it can stack 100 images.
CCDstack has improved with its memory issues but its still very memory intensive.

I agree with Rick the data rejection algorithims must work better with larger samples as they are statistical maths techniques which all have the common denominator of trying to isolate the odd pixel out and reject it. Also median combine is going to work better as well.

Just how many subs before the larger stack improvement weakens to a neglible point I don't know. At a guess I imagine its around 10 to 15 subs. I'll have to ask Stan Moore about that. I doubt that needing 30 or 50 or even 100 making a difference is correct but I don't know for sure.

Greg.