Go Back   IceInSpace > Equipment > Astrophotography and Imaging Equipment and Discussions

Reply
 
Thread Tools Rate Thread
  #1  
Old 19-06-2016, 04:22 PM
andyc's Avatar
andyc (Andy)
Registered User

andyc is offline
 
Join Date: Dec 2011
Location: Sydney
Posts: 1,003
Planetary imaging raw data - what do you keep?

I'm curious to know what other planetary imagers do for keeping raw data files from planetary imaging sessions? I currently have some hard drives groaning with nearly 2TB of data (including both raw and processed) from this year's season of imaging. As I'm quite new to this, it's taking me aback a bit. Some of the raw *.ser files are >10GB (cropped area around Mars, 20k frames, for example). I preprocess in PIPP to debayer and select the best 2-4000 frames (depending on raw stack size, more or less), saving as *.avi, and that can be a 2-3GB file even if the frame's trimmed a bit more around the planet. Final images are small after Registax! But one night's data might be 50GB for a number of images of one planet?

So the question is, being that I'm from a science background and hate throwing away raw data , I'm a bit wary of just keeping the PIPP AVI (and throwing the .ser), but that seems the best approach for not ending up with piles of hard drives . I'm sort of afraid I might one day learn good new techniques and then find I don't have the raw data to use it on, even if that's quite unlikely. All this isn't an issue for deep sky, there's way less data to worry about!

So what do other imagers do? Soak up huge raw data piles, or trim the data in some way? What do you keep, what do you discard?
Reply With Quote
  #2  
Old 20-06-2016, 09:22 AM
Camelopardalis's Avatar
Camelopardalis (Dunk)
Drifting from the pole

Camelopardalis is offline
 
Join Date: Feb 2013
Location: Brisbane
Posts: 5,425
Andy, I keep the originals, so as not to destroy the original timecode. Hard drives are relatively cheap
Reply With Quote
  #3  
Old 21-06-2016, 07:21 AM
sil's Avatar
sil (Steve)
Not even a speck of dust

sil is offline
 
Join Date: Jun 2012
Location: Canberra
Posts: 1,474
A NAS and I keep all source too. When capturing planetary I reduce capture dimension to something smaller (ie crop at source), this reduces file sizes and I get faster frame rates too.

If filesizes are a real problem for you maybe look into lossless codecs for capture and do some testing to see if you are confident they dont introduce compression artifacts etc long ago I used to use a hufyuv (spelling???) lossless codec for recording vhs mostly in general which suited my needs for editing/processing to DVD. There are tons of codecs out there so test a few and see which gives you smaller source files for no quality loss, could help your dilemma.

Always keep my source data but delete the intermediate working files, but keep the first single file containing all stacked data, before wavelets etc processing. So if I want to reprocess later on with improved skills, new software/techniques I have source data to start with or source stacked data, whatever I need and saves time as a starting point.
Reply With Quote
  #4  
Old 21-06-2016, 08:45 AM
andyc's Avatar
andyc (Andy)
Registered User

andyc is offline
 
Join Date: Dec 2011
Location: Sydney
Posts: 1,003
Thanks sil and Dunk, I appreciate the comments. Sil, I already crop pretty well with firecapture, so that part is covered. Recently I experimented with the real-time tighter cropping that firecapture can do, could be good for Jupiter and Saturn, but it does limit the framerate a bit. I'll look into the codecs too, and not keeping the intermediates is also a decent idea. Otherwise, hard drives it is! (just bought another )
Reply With Quote
  #5  
Old 21-06-2016, 01:32 PM
sil's Avatar
sil (Steve)
Not even a speck of dust

sil is offline
 
Join Date: Jun 2012
Location: Canberra
Posts: 1,474
Seriously a NAS is what you want. I have a large tub filled with now empty external drives and keep finding more drives in my computer room to dump to my NAS, can seem costly up front and get one with a bunch of slots and can handle 8TB drives and maybe has an expansion option too. I now have a dozen 2TB drives sitting on my shelf useless to me, they are too small to bother using. With a NAS you can add drives later as you save money or they come down in price and they have great features for setting up your storage space and hybrid RAID configurations for speed and data safety.
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 09:09 AM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Advertisement
Bintel
Advertisement
Testar
Advertisement