Go Back   IceInSpace > Equipment > Software and Computers
Register FAQ Calendar Today's Posts Search

Reply
 
Thread Tools Rate Thread
  #1  
Old 20-07-2015, 09:10 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
DSLR RAW data reduction script - Linux - video demo

I may have posted this in the wrong section. DIY, but software. Apologies, forgot about that rule!

New version bottom of this post - minor fix.

http://www.iceinspace.com.au/forum/s...d.php?t=136926

I have been at this for several months and notice others doing similar things. While there are Linux astro tools, I decided on the command line capability of dcraw and ImageMagick to perform the reduction task and zenity as a GUI.

It is a bash script and essentially a reduction process for DSLR RAW data - files need to be imported to other programs for deBayering, alignment and stacking. Installation is optional. It can be run from the extracted folder. It will ask the user to nominate or create a new directory for the process.

There is a deBayering, alignment and stacking version of the script using a self similarity interpolation algorithm, but it is 8 bit, terribly slow and proprietary and cant be used otherwise - proof of concept - it works.

align_image_stack does just what its name implies. I would like to use the libdc1394 debayer library for 16 bit interpolation, but not sure how to implement this into bash. I did try bayer2rgb, which is based on the libdc1394 bayer.c library, but the output is screwy, not supported from what I can see and will take a lot of expert work to get working.

My scripting and C++ skills are improving but I am working through the tutorials as I write the code.
Attached Files
File Type: zip rawprepro_2.2.zip (19.0 KB, 29 views)

Last edited by rcheshire; 01-08-2015 at 11:19 PM. Reason: Added file
Reply With Quote
  #2  
Old 20-07-2015, 10:24 AM
glend (Glen)
Registered User

glend is offline
 
Join Date: Jun 2013
Location: Lake Macquarie
Posts: 7,121
Way over my head Rowland, but I enjoy reading about what your up to now.

Is there a way to explain what your trying to achieve in more basic (layman) terms?
Reply With Quote
  #3  
Old 20-07-2015, 11:10 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Quote:
Originally Posted by glend View Post
Way over my head Rowland, but I enjoy reading about what your up to now.

Is there a way to explain what your trying to achieve in more basic (layman) terms?
I thought I was explaining it in laymans terms! That explains a thing or two!

This is the theory - DSLR RAW data is heavily modified by camera firmware, and there is no way of turning it off. The camera manufacturer needs to present an image to the user that looks like a photo. Consequently, the linear data from the sensor is modified to look good - this is particularly true of light and dark frames. To some extent bias and flats are more linear, or so I am informed.

The problem arises where the bias is subtracted from both the dark and light frames, sometimes with a view to scaling data. Unfortunately, this can lead to truncating/loosing data. A better method, certainly for beginners, is to leave the bias in the dark.

This is not new by any means - to which "H" will attest, as he has reminded me and others on numerous occasions that the bias is in the dark, and better left there. He is correct, and Jim Solomon nicely presents the formula in his Astrophotography Cookbook.

I think, in some cases DSLR RAW data reduction/calibration has been hijacked - well meant with unintended consequences - by linear 16 bit data reduction methods, obliterating the finer detail available in DSLR RAW data, properly processed.

The script sets out a folder structure and manipulates the images as follows;

1. RAW to 16bit tiff conversion with dcraw; then using ImageMagick
2. Average combine bias and dark frames to produce master bias and master dark frames.
3. Subtract the master bias from flats and median combine the flats to produce a master flat.
4. Subtract the master dark from the light frames and divide by the master flat.

EDIT: If you have dark flats substitute these for bias frames - as they are intended for flat reduction.

Cooled - sub-zero application.

There is also a no dark frame option for cooled DSLRs. Selected, the script will not prompt for, or process, dark frames. Instead it uses the bias, quite contrary to what I said above. But as you know there is little to differentiate darks and bias frames at sub-zero temperatures - so I think its OK.

If you wanted to take flats at a lower ISO, you will take separate sets of bias frames for flat and light frame subtraction. In that case load the bias frames intended for the lights as dark frames an those intended for the flats as bias.

If you have a dark library, (providing the dark is not bias subtracted) just copy your master frames to the image reduction directory and rename per the convention; e.g., MDARK_project_name.tiff - then 'Skip" all of the processes you don't need.

Interesting exercise and lots learned. I have reprocessed pretty much all archived stuff and seen first hand the improvement in image quality.

Caveat. Temperature consistency between lights and darks is a prerequisite. I will get around to scaling data.

Last edited by rcheshire; 01-08-2015 at 11:20 PM.
Reply With Quote
  #4  
Old 23-07-2015, 05:51 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Here is an earlier command line version. I am informed it works in MacOS Yosmite as well as Linux.
Attached Files
File Type: zip rawprepro_beta_0.8208.zip (242.6 KB, 30 views)

Last edited by rcheshire; 30-07-2015 at 08:25 PM.
Reply With Quote
  #5  
Old 01-08-2015, 03:52 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Video - no sound

Demo video - no sound

EDIT: Converts a variety of popular RAW formats - CR2 cr2 CRW NEF nef RAF raf and others or on request.

Script produces 3 outputs - this is explained in documentation supplied with program.

1. 16 bit integer CFA monochrome.tiff - these can be used in a variety of software. In particular, StarTools recommends this for loading into DSS.

What does all that mean?

The 12 or 14bit DSLR RAW file is converted to a 16 bit depth file with the bayer or other color filter array intact - the file is monochrome to save memory and speed up calibration.

2. Master bias, dark and flat frames to use in any program or to calibrate the light frames with the script.

3. Calibrated 16bit CFA monochrome.tiff light frames ready for debayer, alignment and stacking.

Last edited by rcheshire; 08-08-2015 at 09:50 AM.
Reply With Quote
  #6  
Old 05-09-2015, 11:21 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Latest version further improvements to handling flat frames

v2.7 and CL 1.2

This version has improved handling of variations in flat luminosity and avoids under or over correction during flat division of the light frames.

Cores should not be blown out and there should be no signs of binarization, which is not unusual with bad flats or improperly calibrated flats.

All other versions deprecated.

GUI Linux version.

rawprepro_2.7

Command line version for Linux and MacOS.

rawprepro_cl_1.2

Video
Reply With Quote
  #7  
Old 07-05-2016, 12:21 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Latest version

Latest versions here. Flat division is sorted but good flats are required. Dodgy flats 'might' work with the no -auto-level version.
Reply With Quote
  #8  
Old 07-05-2016, 02:20 PM
glend (Glen)
Registered User

glend is offline
 
Join Date: Jun 2013
Location: Lake Macquarie
Posts: 7,121
Love your dedication to the dslr solutions Rowland. I admit i don't understand much of it but i will agree whole heartedly that dithering is magic, it certainly has allowed cleanup processing of my mono sensor faults.
Reply With Quote
  #9  
Old 07-05-2016, 05:41 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Happy to share Glen. Remarkable what you can do with free software like dcraw and imagemagick and the many other programs. Just need a good free debayer algorithm... hmmm

Good old dithering solves a lot of problems, don't know what we'd do without it.
Reply With Quote
  #10  
Old 08-05-2016, 05:12 PM
luka's Avatar
luka
Unregistered User

luka is offline
 
Join Date: Apr 2007
Location: Perth, Australia
Posts: 1,164
Hi Rowland

Very interesting, thank you for sharing. I tried to use it (and to understand the process) but ran into problems.

Firstly can I quickly summarise what I think it is doing, omitting all the fine details:
1. Copy Lights (L), Flats (F), Bias (B) and Darks (D) to separate folders
2. Crate TIFFs for each of them in folders lightgray_...
3. Create master TIFFs for F, B and D in the main folder

At this point I noticed that the master flat was not created. This is where I am stuck. Any ideas.

To keep going:
4. Calculate lights. I suppose this did not work for me because the master flat was not created.
5. Create align and debayer folders.

Would the next step be to stack the individual calibrated lights only (with DSS for example) without need to use D, F or B files. And then post-processing?
Reply With Quote
  #11  
Old 08-05-2016, 07:06 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Hi Luka.

Did the raw flats load?

I have been using it over the weekend without any problems at all. Please find a replacement zip file attached...

Basically the process is as you describe. The raw files are moved to their respective folders to avoid taking up disk space unnecessarily.

Temporary folders are created and cleaned up after processing to save disk space.

The _gray_project_name folder contains the 16bit tiff files - StarTools users have told me they prefer a 16bit tiff for loading into DSS, rather than their raw files.

The raw conversion ensures the tiff data is as representative of the raw data as possible.

Load the calibrated lights into DSS/another program for debayering, alignment and stacking. If that's problematic use the full set of B D F and L converted tiffs in the _grey... folder as you would loading your raw files first up.

EDIT: If you only want converted tiffs you can skip out of the rest of the process.
Attached Files
File Type: zip rawprepro_2.10.zip (20.1 KB, 18 views)
Reply With Quote
  #12  
Old 08-05-2016, 10:37 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
rawprepro_2.10 is written for later versions of ImageMagick. If your distro comes with an older version I recommend updating with the latest ppa. Some of the IM syntax has changed.
Reply With Quote
  #13  
Old 08-05-2016, 11:45 PM
luka's Avatar
luka
Unregistered User

luka is offline
 
Join Date: Apr 2007
Location: Perth, Australia
Posts: 1,164
Hi Rowland

I have found the problem but was unable to solve the issue. The code as it is runs only if installed and not if run from the folder without installation. Can you try uninstalling it and running it from a folder? Which shell are you using? I am using bash which may be the cause of my problems.

So, the problem (at least with bash) is in the if function in two files. For example, in rawprepro you have:
# location of batch_convert
if [ /usr/local/bin/batch_convert ]; then
BATCH=$(readlink -f /usr/local/bin/batch_convert); else
BATCH=$(readlink -f pwd/batch_convert)
fi

The if syntax is not correct and will always evaluate to true. The corrected version should be (note the bold parts):
# location of batch_convert
if [ -e /usr/local/bin/batch_convert ]; then
BATCH=$(readlink -f /usr/local/bin/batch_convert); else
BATCH=$(readlink -f $PWD/batch_convert)
fi

Note also that pwd did not work for me but $PWD does.

Similarly batch_convert has the same issue in two places.

However, this does not solve the problem as $PWD in batch_convert returns the project folder and not the folder in which rawprepro scripts are (rawprepro changes directory before calling batch_convert).

Does this make sense?
Reply With Quote
  #14  
Old 09-05-2016, 01:30 AM
luka's Avatar
luka
Unregistered User

luka is offline
 
Join Date: Apr 2007
Location: Perth, Australia
Posts: 1,164
Hi Rowland

I tried stacking a set of calibrated lights with DSS but I could not figure out how to make DSS do the debayering. It does it automatically for RAW files. What am I missing? I am very new to all this processing (it used to be RAW -> DSS -> curves in PS).

Thanks
Luka
Reply With Quote
  #15  
Old 09-05-2016, 05:51 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Hi Luka.

It is written in bash and I am successfully running the script from the extracted folder as well as installed. You are the first person to provide feedback in the field as it were and I really appreciate the effort. Thank you.

I will have a look at the syntax tonight.

I wondered whether DSS would play ball. I will have a look at it tonight. Regim is an alternative and a bit more flexible than DSS. It will also let you process DSLR images in the same way as rawprepro.

Last edited by rcheshire; 09-05-2016 at 07:17 AM.
Reply With Quote
  #16  
Old 09-05-2016, 11:39 AM
luka's Avatar
luka
Unregistered User

luka is offline
 
Join Date: Apr 2007
Location: Perth, Australia
Posts: 1,164
Hi Rowland

Happy to provide feedback. I am learning a lot and even you mentioning a name of another processing software gets me a step forward so thank you for that and for sharing the software.

By the way, don't forget to uninstall the software before running it from the extracted folder. Bin there, done that

And another bit of feedback, would it be possible to pick all the options first (and store them in variables) and then to do all the processing at the end. At the moment when doing lots of large files there are 3 rounds of waiting with selecting options in between.
In other words while waiting I am having 3 small coffees now instead of 1 large coffee
Reply With Quote
  #17  
Old 09-05-2016, 04:56 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
One large coffee sounds good.

I basically learned bash scripting while writing rawprepro so my skills are amateur at best. That's a good idea but I would need to ensure that the conversion only option is retained for those who just need that.
Reply With Quote
  #18  
Old 10-05-2016, 06:35 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
The latest version with changes can be found here.

Will explore a menu style format over the coming weeks, making it possible to select the desired options and press GO... sit back and wait.
Reply With Quote
  #19  
Old 15-05-2016, 10:28 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
v2.12 - preselect processes

The latest version with changes can be found here - V2.12.

All processes can be preselected or not selected and then left to complete...

Luka you can have that long coffee now.

I have only rejigged the sequence of the zenity dialogs - a bit lame but working on what seems to be the more complex process of getting the zenity checklist options to work.
Reply With Quote
  #20  
Old 16-05-2016, 01:13 AM
luka's Avatar
luka
Unregistered User

luka is offline
 
Join Date: Apr 2007
Location: Perth, Australia
Posts: 1,164
Great Rowland, thank you. I will try to find time to give it a go tomorrow. I started working on the cold finger mod for the 450D which is taking most of my spare time (which is unfortunately close to non-existent ATM).

I was also thinking how to "fix" the DSS so it does not subtract the bias from dark and light frames:
1. Load all flats and all bias frames. Also load 2 light frames (any files will do, DSS will refuse to proceed without any light frames loaded). Register the images, do not stack.
2. Clear the file list.
3. Load the master flat created in step 1 and load all lights and darks. Do not load the bias files.
4. Stack.

At point 1 we created the master flat with the bias subtracted. Then we use this master flat and do not use the bias files any more so they are not subtracted from the lights or from the darks.

Would that work?
Reply With Quote
Reply

Bookmarks


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 01:38 AM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Advertisement
Astrophotography Prize
Advertisement
Bintel
Advertisement