Go Back   IceInSpace > Equipment > Software and Computers

Reply
 
Thread Tools Rate Thread
  #1  
Old 11-06-2017, 02:13 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Linux image preprocessing - Asterism

I have been working on a Linux preprocessing utility for about two years, primarily for my own use as an educational tool and the essential ingredients, curiosity and fun. Now in its fourth iteration, it is a more functional package with a new name features and interface.

Noted: there are plenty of excellent preprocessing programs out there... and I use PixInsight for many tasks. I am just sharing what I have been doing in astrophotography - DIY software.

Primarily intended for consumer digital camera RAW data, the Asterism user manual explains far more than I should include here.

Asterism is a bash script that performs a number of tasks more clearly arranged in a tabbed user interface - compared to previous versions.

Asterim uses existing programs to do the preprocessing work. Yad (GUI) ImageMagick (Q16) and Panotools are required. Hugin MacroFusion and Enfuse are optional. Note: libtiff should be installed on most Linux distros - if not it is also a requirement.

Asterism's primary function is preprocessing. That is image reduction / calibration deBayer alignment and stacking. Secondary processes are stand-alone image calibration, without alignment and stacking. Master frame creation. Conversion (16 bit monochrome Bayer) - not that useful unless you have special requirements.

There are range of other options, again better explained by viewing the documentation and playing with a small image set to illustrate the various functions.

Not much more to say on the topic. It's a work in progress, but functional, all the same.

Last edited by rcheshire; 15-06-2017 at 09:50 AM.
Reply With Quote
  #2  
Old 19-06-2017, 03:54 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Update: I've added a polynomial stacking routine. Essentially a noise rejection filter. Removes satellite trails and hot pixels. Before and after shots. These are low resolution jpg normalized gamma 2.2 to illustrate the effect.
Attached Thumbnails
Click for full-size image (post-228066-0-87483800-1497825500.jpg)
72.5 KB63 views
Click for full-size image (post-228066-0-04502800-1497825520.jpg)
60.2 KB61 views

Last edited by rcheshire; 27-06-2017 at 01:40 PM.
Reply With Quote
  #3  
Old 24-06-2017, 11:14 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
A new version. No major changes. Largely bug fixes, code tidy up and provision for setting ImageMagick in-built resources - threads and memory - on a per-session basis. Removed unnecessary parallel processing routines, letting ImageMagick handle the resources.

EDIT: with reference to the first post, dcraw is also required

Last edited by rcheshire; 27-06-2017 at 01:44 PM.
Reply With Quote
  #4  
Old 01-07-2017, 07:04 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
New version and manual - minor bug fix. LLRGB has been remodelled and Bayer non-raw images can be processed. Asterism manual refers.
Reply With Quote
  #5  
Old 06-07-2017, 11:05 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Asterism_superbias creates what might be called a superbias, significantly reducing random noise in the master bias
Attached Thumbnails
Click for full-size image (sbias.jpg)
52.5 KB19 views
Click for full-size image (sflat.jpg)
9.6 KB18 views
Reply With Quote
  #6  
Old 08-07-2017, 04:05 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Added a standalone alignment and stacking utility. User manual refers.

EDIT: Documentation update

Last edited by rcheshire; 09-07-2017 at 11:21 AM.
Reply With Quote
  #7  
Old 19-07-2017, 08:15 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
New version.

Last edited by rcheshire; 19-07-2017 at 12:44 PM.
Reply With Quote
  #8  
Old 22-07-2017, 12:54 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Tweaks and bug fix. Super-bias process modified.

EDIT: this process is essentially a large kernel median filter. The larger the kernel the longer the process. Comparing histograms of various kernel sizes, the latest iteration should be a good compromise between noise reduction in the bias and processing time.

Last edited by rcheshire; 25-07-2017 at 06:39 AM.
Reply With Quote
  #9  
Old 31-07-2017, 09:45 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Added channel (matrix) sRGB multiplier capability... user manual refers.

Obtain channel matrix values here, for example

Last edited by rcheshire; 31-07-2017 at 10:19 AM.
Reply With Quote
  #10  
Old 17-08-2017, 06:49 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Hot pixel map correction

This latest version includes hot pixel map correction. Certainly removes hot pixels. Flat illumination correction method reviewed and simplified, hopefully. Not to mention a significant manual rewrite.

The interface has seen change as well as the inclusion of links to documentation and other resources from the Documentation tab.

EDIT: uploaded a revision - same link.

Last edited by rcheshire; 18-08-2017 at 10:26 AM.
Reply With Quote
  #11  
Old 19-08-2017, 09:37 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Please note that hot pixel map is pretty much testing and I am working on improving the process or abandoning it altogether - it is optional. Polynomial stacking takes care of hot pixels and other outliers very well besides.
Reply With Quote
  #12  
Old 20-08-2017, 08:46 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Flat illumination correction

Changes to user manual description for use of flat illumination correction.
Reply With Quote
  #13  
Old 07-09-2017, 05:55 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Updates

Updates - minor
Reply With Quote
  #14  
Old 21-11-2017, 06:11 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
More changes / improvements

In this version, significant changes.

The manual has been overhauled and should be easier to follow.

The defect map process has been redesigned

Noise reduction may be applied to the final image

The intent has been to find authentic methods of noise reduction which may be applied non-destructively during image preprocessing

I have tested the new processes over and again. The original binarized defect map is still the default. The better (IMHO), mean-shift method, is activated by setting a 'Noise - radius' - '7' is adequate and a 'Noise - amount' - 0.1% at most.

1 or more dark frames may be used to create the defect map; which is really a master dark with considerable noise reduction, leaving the white and black pixels intact. A median dark is a fair substitute.

With mean-shift noise reduction of the final image, combined with polynomial stacking, the statistical handling of data appears authentic and visually non-destructive - pixels retain their shape and there is no evidence of smearing, unless the 'Noise - amount' operator is overdone - 0.1 - 0.25% is adequate.

Last edited by rcheshire; 22-11-2017 at 03:04 PM.
Reply With Quote
  #15  
Old 27-11-2017, 09:38 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Recommended update

Update: A number of issues associated with increased complexity in the previous two versions are resolved.

Asterism_0.0.5.3 is tested current and stable. Upgrade to the latest version is recommended, as it includes a number of improvements;

Flat auto correction - this is proving very robust. The default setting of 5 appears to be ideal. The old manual flat illumination correction is retained, for the time being.

Defect map - the process seems to work very well, with no issues observed - use a ‘Noise - radius’ of 7 and a ‘Noise - amount’ of 0.1. It may be necessary to alter settings for different cameras, although, tested on 3 Canon model RAW data and Nikon, I see no variation, except clean flat division.

Linear noise reduction - this is the noise reduction routine used by Defect map, applied to the stacked linear image.

While Asterism is designed to reduce noise in a number of ways, I have found linear noise reduction very effective. The ‘Noise - amount’ value should not be too high - 0.1

Correctly applied, the stacked image processed in StarTools, with Tracking ‘off’, the default noise reduction is adequate.

A ‘Noise - radius’ size, 0 (off) to 30 is available, with sensible radii between 5 - 15; and ‘Noise - amount’ 0.1. Increase the radius for noisy images.

The User Manual requires an update, however, the differences are noted in the version available for download.
Reply With Quote
  #16  
Old 03-12-2017, 08:27 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Latest version - redesigned interface

I am fairly happy with the latest version, in terms of usability and features. 0.0.5.5 is tested and stable and includes a significant user manual rewrite. This version has a redesigned 'Processing and options' tab with segregated fields which are easier to navigate. Options layout is better arranged and should be more intuitive, in the order; input and output, main process selection, optional processes, followed by system defaults.

Last edited by rcheshire; 09-12-2017 at 01:17 PM.
Reply With Quote
  #17  
Old 09-12-2017, 01:16 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
New Asterism_beta is tested and stable, run from the Asterism folder in a terminal. Largely cosmetic changes. PMStack may be launched from the Asterism window.

When complete, all processes return the user to the Asterism main window, automatically, to perform other tasks. Asterism may be closed with the 'Quit' button. Otherwise in accordance with the user manual, which needs an update.
Reply With Quote
  #18  
Old 12-12-2017, 07:51 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,617
Latest Asterism stable version with manual rewrite.
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 11:12 AM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Advertisement
Testar
Advertisement
Bintel
Advertisement