DSLR RAW data reduction script - Linux - video demo

[rcheshire]

I may have posted this in the wrong section. DIY, but software. Apologies, forgot about that rule!

New version bottom of this post - minor fix.

http://www.iceinspace.com.au/forum/showthread.php?t=136926

I have been at this for several months and notice others doing similar things. While there are Linux astro tools, I decided on the command line capability of dcraw and ImageMagick to perform the reduction task and zenity as a GUI.

It is a bash script and essentially a reduction process for DSLR RAW data - files need to be imported to other programs for deBayering, alignment and stacking. Installation is optional. It can be run from the extracted folder. It will ask the user to nominate or create a new directory for the process.

There is a deBayering, alignment and stacking version of the script using a self similarity interpolation algorithm, but it is 8 bit, terribly slow and proprietary and cant be used otherwise - proof of concept - it works.

align_image_stack does just what its name implies. I would like to use the libdc1394 debayer library for 16 bit interpolation, but not sure how to implement this into bash. I did try bayer2rgb, which is based on the libdc1394 bayer.c library, but the output is screwy, not supported from what I can see and will take a lot of expert work to get working.

My scripting and C++ skills are improving but I am working through the tutorials as I write the code.

[glend]

Way over my head Rowland, :shrug: but I enjoy reading about what your up to now.:thumbsup:

Is there a way to explain what your trying to achieve in more basic (layman) terms?

[rcheshire]

I thought I was explaining it in laymans terms! That explains a thing or two!

This is the theory - DSLR RAW data is heavily modified by camera firmware, and there is no way of turning it off. The camera manufacturer needs to present an image to the user that looks like a photo. Consequently, the linear data from the sensor is modified to look good - this is particularly true of light and dark frames. To some extent bias and flats are more linear, or so I am informed.

The problem arises where the bias is subtracted from both the dark and light frames, sometimes with a view to scaling data. Unfortunately, this can lead to truncating/loosing data. A better method, certainly for beginners, is to leave the bias in the dark.

This is not new by any means - to which "H" will attest, as he has reminded me and others on numerous occasions that the bias is in the dark, and better left there. He is correct, and Jim Solomon nicely presents the formula in his Astrophotography Cookbook.

I think, in some cases DSLR RAW data reduction/calibration has been hijacked - well meant with unintended consequences - by linear 16 bit data reduction methods, obliterating the finer detail available in DSLR RAW data, properly processed.

The script sets out a folder structure and manipulates the images as follows;

1. RAW to 16bit tiff conversion with dcraw; then using ImageMagick
2. Average combine bias and dark frames to produce master bias and master dark frames.
3. Subtract the master bias from flats and median combine the flats to produce a master flat.
4. Subtract the master dark from the light frames and divide by the master flat.

EDIT: If you have dark flats substitute these for bias frames - as they are intended for flat reduction.

Cooled - sub-zero application.

There is also a no dark frame option for cooled DSLRs. Selected, the script will not prompt for, or process, dark frames. Instead it uses the bias, quite contrary to what I said above. But as you know there is little to differentiate darks and bias frames at sub-zero temperatures - so I think its OK.

If you wanted to take flats at a lower ISO, you will take separate sets of bias frames for flat and light frame subtraction. In that case load the bias frames intended for the lights as dark frames an those intended for the flats as bias.

If you have a dark library, (providing the dark is not bias subtracted) just copy your master frames to the image reduction directory and rename per the convention; e.g., MDARK_project_name.tiff - then 'Skip" all of the processes you don't need.

Interesting exercise and lots learned. I have reprocessed pretty much all archived stuff and seen first hand the improvement in image quality.

Caveat. Temperature consistency between lights and darks is a prerequisite. I will get around to scaling data.

[rcheshire]

Here is an earlier command line version. I am informed it works in MacOS Yosmite as well as Linux.

[rcheshire]

Demo video - no sound (http://synergous.com/flatpress/fp-content/attachs/rawprepro.mp4)

EDIT: Converts a variety of popular RAW formats - CR2 cr2 CRW NEF nef RAF raf and others or on request.

Script produces 3 outputs - this is explained in documentation supplied with program.

1. 16 bit integer CFA monochrome.tiff - these can be used in a variety of software. In particular, StarTools recommends this for loading into DSS.

What does all that mean?

The 12 or 14bit DSLR RAW file is converted to a 16 bit depth file with the bayer or other color filter array intact - the file is monochrome to save memory and speed up calibration.

2. Master bias, dark and flat frames to use in any program or to calibrate the light frames with the script.

3. Calibrated 16bit CFA monochrome.tiff light frames ready for debayer, alignment and stacking.

[rcheshire]

v2.7 and CL 1.2

This version has improved handling of variations in flat luminosity and avoids under or over correction during flat division of the light frames.

Cores should not be blown out and there should be no signs of binarization, which is not unusual with bad flats or improperly calibrated flats.

All other versions deprecated.

GUI Linux version.

rawprepro_2.7 (http://synergous.com/flatpress/fp-content/attachs/rawprepro_27.zip)

Command line version for Linux and MacOS.

rawprepro_cl_1.2 (http://synergous.com/flatpress/fp-content/attachs/rawprepro_cl_12.zip)

Video (http://synergous.com/flatpress/fp-content/attachs/rawprepro.mp4)

[rcheshire]

Latest versions here (http://www.synergous.com/flatpress/index.php/2014/05/01/astrophotography-dslr-raw-data-calibration-preprocessing/). Flat division is sorted but good flats are required. Dodgy flats 'might' work with the no -auto-level version.

[glend]

Love your dedication to the dslr solutions Rowland. I admit i don't understand much of it but i will agree whole heartedly that dithering is magic, it certainly has allowed cleanup processing of my mono sensor faults.

[rcheshire]

Happy to share Glen. Remarkable what you can do with free software like dcraw and imagemagick and the many other programs. Just need a good free debayer algorithm... hmmm

Good old dithering solves a lot of problems, don't know what we'd do without it.

[luka]

Hi Rowland

Very interesting, thank you for sharing. I tried to use it (and to understand the process) but ran into problems.

Firstly can I quickly summarise what I think it is doing, omitting all the fine details:
1. Copy Lights (L), Flats (F), Bias (B) and Darks (D) to separate folders
2. Crate TIFFs for each of them in folders lightgray_...
3. Create master TIFFs for F, B and D in the main folder

At this point I noticed that the master flat was not created. This is where I am stuck. Any ideas.

To keep going:
4. Calculate lights. I suppose this did not work for me because the master flat was not created.
5. Create align and debayer folders.

Would the next step be to stack the individual calibrated lights only (with DSS for example) without need to use D, F or B files. And then post-processing?

[rcheshire]

Hi Luka.

Did the raw flats load?

I have been using it over the weekend without any problems at all. Please find a replacement zip file attached...

Basically the process is as you describe. The raw files are moved to their respective folders to avoid taking up disk space unnecessarily.

Temporary folders are created and cleaned up after processing to save disk space.

The _gray_project_name folder contains the 16bit tiff files - StarTools users have told me they prefer a 16bit tiff for loading into DSS, rather than their raw files.

The raw conversion ensures the tiff data is as representative of the raw data as possible.

Load the calibrated lights into DSS/another program for debayering, alignment and stacking. If that's problematic use the full set of B D F and L converted tiffs in the _grey... folder as you would loading your raw files first up.

EDIT: If you only want converted tiffs you can skip out of the rest of the process.

[rcheshire]

rawprepro_2.10 is written for later versions of ImageMagick. If your distro comes with an older version I recommend updating with the latest ppa. Some of the IM syntax has changed.

[luka]

Hi Rowland

I have found the problem but was unable to solve the issue. The code as it is runs only if installed and not if run from the folder without installation. Can you try uninstalling it and running it from a folder? Which shell are you using? I am using bash which may be the cause of my problems.

So, the problem (at least with bash) is in the if function in two files. For example, in rawprepro you have:
# location of batch_convert
if [ /usr/local/bin/batch_convert ]; then
BATCH=$(readlink -f /usr/local/bin/batch_convert); else
BATCH=$(readlink -f pwd/batch_convert)
fi

The if syntax is not correct and will always evaluate to true. The corrected version should be (note the bold parts):
# location of batch_convert
if [ -e /usr/local/bin/batch_convert ]; then
BATCH=$(readlink -f /usr/local/bin/batch_convert); else
BATCH=$(readlink -f $PWD/batch_convert)
fi

Note also that pwd did not work for me but $PWD does.

Similarly batch_convert has the same issue in two places.

However, this does not solve the problem as $PWD in batch_convert returns the project folder and not the folder in which rawprepro scripts are (rawprepro changes directory before calling batch_convert).

Does this make sense?

[luka]

Hi Rowland

I tried stacking a set of calibrated lights with DSS but I could not figure out how to make DSS do the debayering. It does it automatically for RAW files. What am I missing? I am very new to all this processing (it used to be RAW -> DSS -> curves in PS).

Thanks
Luka

[rcheshire]

Hi Luka.

It is written in bash and I am successfully running the script from the extracted folder as well as installed. You are the first person to provide feedback in the field as it were and I really appreciate the effort. Thank you.

I will have a look at the syntax tonight.

I wondered whether DSS would play ball. I will have a look at it tonight. Regim (http://www.andreasroerig.de/regim/regim_e.htm) is an alternative and a bit more flexible than DSS. It will also let you process DSLR images in the same way as rawprepro.

[luka]

Hi Rowland

Happy to provide feedback. I am learning a lot and even you mentioning a name of another processing software gets me a step forward so thank you for that and for sharing the software.

By the way, don't forget to uninstall the software before running it from the extracted folder. Bin there, done that :)

And another bit of feedback, would it be possible to pick all the options first (and store them in variables) and then to do all the processing at the end. At the moment when doing lots of large files there are 3 rounds of waiting with selecting options in between.
In other words while waiting I am having 3 small coffees now instead of 1 large coffee :D

[rcheshire]

One large coffee sounds good.

I basically learned bash scripting while writing rawprepro so my skills are amateur at best. That's a good idea but I would need to ensure that the conversion only option is retained for those who just need that.

[rcheshire]

The latest version with changes can be found here. (http://synergous.com/flatpress/index.php/2014/05/01/astrophotography-dslr-raw-data-calibration-preprocessing/)

Will explore a menu style format over the coming weeks, making it possible to select the desired options and press GO... sit back and wait.

[rcheshire]

The latest version with changes can be found here - V2.12. (http://synergous.com/flatpress/index.php/2014/05/01/astrophotography-dslr-raw-data-calibration-preprocessing/)

All processes can be preselected or not selected and then left to complete...

Luka you can have that long coffee now.

I have only rejigged the sequence of the zenity dialogs - a bit lame but working on what seems to be the more complex process of getting the zenity checklist options to work.

[luka]

Great Rowland, thank you. I will try to find time to give it a go tomorrow. I started working on the cold finger mod for the 450D which is taking most of my spare time (which is unfortunately close to non-existent ATM).

I was also thinking how to "fix" the DSS so it does not subtract the bias from dark and light frames:
1. Load all flats and all bias frames. Also load 2 light frames (any files will do, DSS will refuse to proceed without any light frames loaded). Register the images, do not stack.
2. Clear the file list.
3. Load the master flat created in step 1 and load all lights and darks. Do not load the bias files.
4. Stack.

At point 1 we created the master flat with the bias subtracted. Then we use this master flat and do not use the bias files any more so they are not subtracted from the lights or from the darks.

Would that work?

[rcheshire]

I'm not sure but that's an innovative solution... if DSS plays ball - worth a try.

It might be easier to convert your raw files and create your masters with the script and drop them into DSS with no bias frames or master.

Regim, which is free software, will do all of this without manipulation, but I have found that getting the settings right is essential to success. If you go to the StarTools site there's a post in the discussion forum on image calibration with Regim which might help should you give it a go.

[luka]

Hi Rowland

The trick with the DSS may be useful for people who don't run linux. Tricking DSS worked (and DSS complained about the lack of bias files in the last step) but I am not sure if it did the right thing.

Anyway, my problem with the master TIFF files created by your script was debayering before stacking. DSS will not debayer TIFF files (at least I could not get it to do it).

I tried Regim a few days ago and liked it a lot (thank you for the hint :thumbsup:). More linux programs will make my life easier :-)

Regim will debayer TIFF files but, as far as I could see, it will only do one file at a time (menu Color -> RAW debayering). Ticking "Files are RAW" in Preprocessing -> calibration works only for RAW files, TIFF files were not debayered. I will check the forum you mentioned.

How are you actually debayering files from your DSLR? Am I missing an option in Regim (very likely).

Again thank you
Luka

[rcheshire]

I use PixInsight for debayering, alignment and stacking but prefer to use my script for calibrating DSLR files...

Regim startools discussion - here. (http://forum.startools.org/viewtopic.php?f=10&t=752&p=3140&hilit=regim#p3140) Elaborates the settings. It had me thinking for a while - not easy to use. You should be able to load sets of files rather than one at a time. I will have another look tonight. But I think the thread covers it.

...still working on a debayer function to be followed by an alignment process - WIP.

[luka]

Hi Rowland

Thank you for the tip. I tried again to debayer multiple images (in preprocessing) with regim and failed. Then I thought I may try clearing the settings so I deleted the folder and started fresh - it worked straight away. No idea what I did differently, the same tick boxes were ticked (100% sure).

I have not tried your latest version yet, I run out of coffee :-)
Possibly tonight...

[rcheshire]

Here (http://www.synergous.com/flatpress/index.php/2014/05/01/astrophotography-dslr-raw-data-calibration-preprocessing/)

Fixed bugs in install and remove scripts - added licence agreement.

[rcheshire]

Now with process preselection check boxes - includes an option to purge (nuke) the unprocessed RAW to TIFF converted 16bit files if these are not required - the check box is hidden from sight and is not selected by default. Need to scroll down the window to find it.

Note: bias flat and light frame selection windows are displayed by default as the minimum requirement (with cooled DSLRs in mind). Dark frames are optional but selected by default. All the windows may be cancelled without complication if RAW files were loaded, for instance, during a previous run.

The program may be run in user space and in that case will not write files to the system, otherwise. If installed, rawprepro and dependencies may be found in /usr/local/bin

[rcheshire]

This version extends frame selection to all frames...

[rcheshire]

This is the latest...

[rcheshire]

Latest and possibly the last... (http://www.synergous.com/flatpress/index.php/2014/05/01/astrophotography-dslr-raw-data-calibration-preprocessing/)

[rcheshire]

Famous last words...

While this may be of limited interest to many, I have revamped and renamed my Linux preprocessing script (http://synergous.com/flatpress/index.php/2016/11/11/free-astronomical-image-calibration-and-stacking-faics/) (FAICS) to include a RAW to 16 bit monochrome CFA converter and a RAW to 16 bit monochrome CFA calibration routine (previously rawprepro). Both convert to any extension that ImageMagick can handle (fits is an obvious choice - tiff is default). The basic RGB preprocessing - calibration align and stack - script remains in place with some added tweaks. Debayer prior to calibration is not ideal, but it produces 'reasonable' images.

Still working on existing but difficult to implement debayer routine and Sigma rejection algorithm, which remain a very large hurdle with my limited C++ skills.

This is a labour of love and personal learning experience - I realise there are many other very worthy programmes out there.