Go Back   IceInSpace > Equipment > Astrophotography and Imaging Equipment and Discussions

Reply
 
Thread Tools Rating: Thread Rating: 2 votes, 5.00 average.
  #41  
Old 20-05-2014, 03:34 PM
Asterix2020's Avatar
Asterix2020 (Paul)
Registered User

Asterix2020 is offline
 
Join Date: Oct 2013
Location: Delaneys Creek, Qld
Posts: 184
Humayun,
That makes sense that Canon's RAW converter in DPP should give then "best" TIFF files. I don't really know enough to have an opinion though.

I just converted a RAW to TIFF with DPP. The dcraw version is a lot darker compared to the DPP TIFF. What that means I don't know?

The attachment is a single sub, in TIFF, the left converted by DPP and the right by dcraw.
Attached Thumbnails
Click for full-size image (tiff comp.jpg)
195.0 KB32 views
Reply With Quote
  #42  
Old 20-05-2014, 03:37 PM
Octane's Avatar
Octane (Humayun)
IIS Member #671

Octane is offline
 
Join Date: Dec 2005
Location: Canberra
Posts: 11,159
It is proprietary software written by the makers of the hardware themselves.

Everything else is reverse-engineered by using SDKs. All the results that are produced are approximations of what the sensor actually captured.

H
Reply With Quote
  #43  
Old 20-05-2014, 03:38 PM
Asterix2020's Avatar
Asterix2020 (Paul)
Registered User

Asterix2020 is offline
 
Join Date: Oct 2013
Location: Delaneys Creek, Qld
Posts: 184
Barry,
I'm using DDS 3.3.4 - it used dcraw 9.20.
I used dcraw 9.17 for my conversions.
I think it is the choice of parameters. I've asked Ivo to join in this thread has he can probably answer your question much better. I've just followed his advice and have seen better results from it.

Last edited by Asterix2020; 20-05-2014 at 04:27 PM.
Reply With Quote
  #44  
Old 20-05-2014, 04:25 PM
irwjager's Avatar
irwjager (Ivo)
Registered User

irwjager is offline
 
Join Date: Apr 2010
Location: Melbourne
Posts: 490
Quote:
Originally Posted by Octane View Post
Exporting TIFFs from Canon's Digital Photo Professional will give you, in your TIFF, exactly what the sensor captured.
And this is exactly the problem - this is unfortunately the very opposite of the truth and is the reason for all this tomfoolery with dcraw.

For AP we are interested in what the sensor captured, which is *definitely *not* what DPP outputs. DPP just makes things 'pretty', but it's the very opposite of what we need for AP!

1. For starters, Canon DSLRs (as do many other cameras) have a Color Filter Array, meaning that the pixels that are recorded by the CCD are divvied up between the different colors. In the case of your Canon, every 4 native pixels are divided up into 1 red, 2 green and 1 blue channel pixel. E.g. for a 16MP camera, you will get 4MP of red and blue data and 8MP of green data.

This is obviously not the case when you look at a full frame from DPP; all the channels' pixels are accounted for. This is thanks to an interpolation (debayering) algorithm that makes up the missing 3 pixels for red and blue and the missing 2 pixels for the green channel. Yes, that's right - a quarter of your blue and red and half of your green data is completely made up! There are various clever ways of making up this data and chances are that DPP uses a very clever (proprietary) algorithm indeed that reconstructs as much as possible. The trouble is that 'clever' is *bad* for AP. Whereas this works great for terrestrial daytime images with plenty of signal and plenty of geometrical shapes to derive clues from for reconstruction, frames shot for AP purposes have neither. What happens then is that the reconstruction algorithms start reconstructing things that aren't there due to noise and the absence of geometrical shapes/patterns. Or more generally speaking, the noise that was contained in one pixel (for red and blue) is propagated into its neighbouring pixels. What happens then is the creation of artifacts (introducing detail) and noise grain that is larger than one pixel, both of which are very hard to get rid of by stacking or noise reduction algorithms, causing these artifacts and noise grain to become very conspicuous.

Best it then to use a 'dumb' interpolation algorithm (such as bilinear interpolation) to make a 'recoverable' guess in case the guess was wrong (e.g. refraining from trying to introduce detail).

Refer to the attached image; 1 is the original scene, 2 is what the CCD actually captured (which is a far cry from the DPP result - and, also, it's been stretched for easy comparison), 3 is what each individual pixel captured when assigned its correct color, 4 is what the scene looks like reconstructed once the missing data has been interpolated (notice artifacts).

2. As you can see from the comparison that Paul posted, it is clear that the data has been stretched by DPP (so it's fit for human consumption). This is *bad* as many processing steps need to be performed when the data is still in the linear domain (for example deconvolution and color calibration).

3. As you can see from the comparison that Paul posted, it is clear that the data has been color balanced by DPP. This is *bad* as (as I posted on the ST forum) the noise in each channel has also scaled by the white balance factors. It has now become impossible for ST to track the noise effectively as the noise level varies significantly between channels.

4. Since the data has now been white balanced and stretched, it is now no longer possible to recover true colour, as re-whitebalancing is no longer possible and light pollution can be subtracted but its (linear!) influence cannot be rebalanced, also owing to he fact that the whitebalancing will have clipped some of the highlights (white balancing is a simple multiplication of the channels by different factors, consequently causing highlights to carry data for some of the channels but not for others).

5. While I cannot be sure about DPP, I know from other manufacturers that a lot more processing is performed than just simple stretching and colour balancing; often noise reduction and sharpening is applied, wreaking havoc on faint signal that could otherwise be recovered by stacking, or further destroying any hope of applying mathematically correct deconvolution. Sony's NEX line of cameras comes with software that is particularly bad in this respect.

Bottom line is, *please* keep your data as virgin as possible (e.g. as close to actual photon counts as possible). If you don't, you will not be able to make effective or correct (in the case of colour management and deconvolution) use of your signal.
Attached Thumbnails
Click for full-size image (225px-Colorful_spring_garden_Bayer.jpg)
33.6 KB29 views

Last edited by irwjager; 20-05-2014 at 04:53 PM.
Reply With Quote
  #45  
Old 20-05-2014, 04:31 PM
Octane's Avatar
Octane (Humayun)
IIS Member #671

Octane is offline
 
Join Date: Dec 2005
Location: Canberra
Posts: 11,159
Ivo,

I get you.

But, for all intents and purposes, the prettiness that DPP does to the files is exactly what I expect to see once an image is captured.

I do understand that there is so much more underlying what is presented in DPP.

And, I totally agree with you that this is fine for terrestrial photography, where we don't necessarily care about calibration or linearity.

Re: noise reduction; this can be disabled in the noise reduction tab. By default, DPP applies an appropriate amount of luminance and chrominance noise reduction to each image, depending on the ISO it was shot at, and, what camera was used. This can be disabled in the camera, or modified in DPP itself.

I guess my main issue is, how to tell which one (decoder) is correct, when they all produce differing output?

Appreciate your post.

Cheers.

H
Reply With Quote
  #46  
Old 20-05-2014, 04:32 PM
Amaranthus's Avatar
Amaranthus (Barry)
Thylacinus stargazoculus

Amaranthus is offline
 
Join Date: Jan 2014
Location: Judbury, Tasmania
Posts: 1,202
Thanks Ivo, great feedback.

What I'd like to know is if it is okay to import my RAWs into Deep Sky Stacker (not DPP - I use a Sony NEX), and then let DSS call up dcraw to do the conversion, or whether I should be doing the dcraw conversion myself, manually..
Reply With Quote
  #47  
Old 20-05-2014, 04:40 PM
irwjager's Avatar
irwjager (Ivo)
Registered User

irwjager is offline
 
Join Date: Apr 2010
Location: Melbourne
Posts: 490
Quote:
Originally Posted by Amaranthus View Post
Thanks Ivo, great feedback.

What I'd like to know is if it is okay to import my RAWs into Deep Sky Stacker (not DPP - I use a Sony NEX), and then let DSS call up dcraw to do the conversion, or whether I should be doing the dcraw conversion myself, manually..
This is a question for the Luc (DSS author). The trouble that I (and many others) have had with DSS is that DSS does not tell dcraw to respect the data as-is and simply do the debayering. DSS lets dcraw (and the user - but that is a whole other beef I have with DSS) do much more that is detrimental to the data.

For the record, PixInsight also uses dcraw and handles things correctly.
Reply With Quote
  #48  
Old 20-05-2014, 04:50 PM
irwjager's Avatar
irwjager (Ivo)
Registered User

irwjager is offline
 
Join Date: Apr 2010
Location: Melbourne
Posts: 490
Quote:
Originally Posted by Octane View Post
Ivo,

I get you.

But, for all intents and purposes, the prettiness that DPP does to the files is exactly what I expect to see once an image is captured.
Of course. The trouble is this 'preview' has severely diminished your ability to make the most of the signal if you use it as your basis. You know that and I know that, but for newbies, after going straight to the brightness/contrast slider and shooting in JPEG, it's AP sin #1.

Quote:
I guess my main issue is, how to tell which one (decoder) is correct, when they all produce differing output?
The most 'correct' decoding is to dispense with the interpolation completely! There are two ways you can do that. One is to forego resolution so that 1 red and blue pixel truly correspond to 1 RGB pixel. The other is to use a method that is similar to drizzling where each pixel in the CFA/bayer filter gets exposed between different frames by shifting the scope a tiny bit. Scroll down to 'RAW images development process' on this page of the DSS website for more info.

Often though, the native resolution of DSLRs is way to high for the optics/seeing combo to record 1 unit of real detail per CCD pixel anyway (unless you do very-wide-field AP of course), so foregoing resolution is what I would typically recommend.
Reply With Quote
  #49  
Old 20-05-2014, 06:33 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,603
Quite a discussion...
Reply With Quote
  #50  
Old 20-05-2014, 06:37 PM
JohnG's Avatar
JohnG (John)
Looking Down From Above

JohnG is offline
 
Join Date: Dec 2005
Location: Cootamundra, NSW
Posts: 1,696
Been excellent Roland, the way it should be..
Reply With Quote
  #51  
Old 20-05-2014, 06:47 PM
nebulosity.'s Avatar
nebulosity. (Jo)
Registered User

nebulosity. is offline
 
Join Date: Jul 2013
Location: Cecil Plains QLD
Posts: 1,224
I can demosaic (debayer) my RAW CR2 files in the Nebulosity and they look very different to what my other photo programs shows them as.

Nebulosity provides a few debayer options: Colour Binning, Bilinear, VNG, PPG, AHG. Which would give me the best results?

Here is the same RAW image opened in (1) Aperture, (2) Nebulosity, (3) and debayered in Nebulosity. (VNG)

Cheers
Jo

EDIT, Added another version, this is from Digital Photo Professional. Also these are all Canon files.
Attached Thumbnails
Click for full-size image (<untitled>.jpg)
167.9 KB17 views
Click for full-size image (CR2 raw.jpg)
87.1 KB15 views
Click for full-size image (debayered.jpg)
68.9 KB13 views
Click for full-size image (<untitled>2,.jpg)
198.3 KB16 views

Last edited by nebulosity.; 20-05-2014 at 07:03 PM.
Reply With Quote
  #52  
Old 20-05-2014, 06:50 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,603
I got more out of that than expected, John. You will have to show me what you have done with the 60CB. I've been thinking it over for some time.
Reply With Quote
  #53  
Old 20-05-2014, 09:00 PM
irwjager's Avatar
irwjager (Ivo)
Registered User

irwjager is offline
 
Join Date: Apr 2010
Location: Melbourne
Posts: 490
Quote:
Originally Posted by nebulosity. View Post
Nebulosity provides a few debayer options: Colour Binning, Bilinear, VNG, PPG, AHG. Which would give me the best results?
Colour Binning is, if I'm not mistaken, the choice to not interpolate at all and obtain an image at a quarter of the native CCD resolution. If you can live with that, this is what I would recommend.

If you don't like that then I'd recommend Bilinear interpolation because this simple scheme introduces the least artifacts. The other schemes are progressively 'clever' and will introduce artifacts that stacking may not be able to overcome.

Quote:
Here is the same RAW image opened in (1) Aperture, (2) Nebulosity, (3) and debayered in Nebulosity. (VNG)
In reality they are probably not so different and merely have different things applied to them;

(1) is really (2) debayered and stretched but (probably) not colour balanced.

(2) is the raw data as it comes from the CCD. If you zoom in you will notice a checker pattern due to the bayer matrix (not visible in the JPEG).

(3) is the raw data from (2) debayerd - the checker pattern should now be gone

(4) is (1) but colour balanced

That's my guess anyway.
Reply With Quote
  #54  
Old 21-05-2014, 06:05 AM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,603
DSLR RAW data, despite how it might be prepared for processing is vulnerable to the manner in which it is first prepared by the camera and, a number of image processing issues that originate at acquisition. Frankly, I think this is where it all begins, good and bad results.

Furthermore, I think software tends to make assumptions about what we want to do with our data. Besides being aware of DSLR RAW data non-linearity issues and how that might be handled (the readon for this thread) it may be time for a close look at what we should do with our data, rather than what happens to it.

I like the idea of using dcraw options to suit, rather than accepting defaults that presume/make false assumptions about our data. Rah, rah, rah... raise the flag...
Reply With Quote
  #55  
Old 22-05-2014, 08:05 AM
irwjager's Avatar
irwjager (Ivo)
Registered User

irwjager is offline
 
Join Date: Apr 2010
Location: Melbourne
Posts: 490
Quote:
Originally Posted by rcheshire View Post
DSLR RAW data, despite how it might be prepared for processing is vulnerable to the manner in which it is first prepared by the camera and, a number of image processing issues that originate at acquisition. Frankly, I think this is where it all begins, good and bad results.

Furthermore, I think software tends to make assumptions about what we want to do with our data. Besides being aware of DSLR RAW data non-linearity issues and how that might be handled (the readon for this thread) it may be time for a close look at what we should do with our data, rather than what happens to it.

I like the idea of using dcraw options to suit, rather than accepting defaults that presume/make false assumptions about our data. Rah, rah, rah... raise the flag...
Hear hear!

Again, I do find some programs that use dcraw internally better than others in this respect, with PixInsight standing out in particular for keeping the data unadulterated as and clean as possible (what it does to it afterwards is up for debate however )
Reply With Quote
  #56  
Old 22-05-2014, 12:05 PM
Amaranthus's Avatar
Amaranthus (Barry)
Thylacinus stargazoculus

Amaranthus is offline
 
Join Date: Jan 2014
Location: Judbury, Tasmania
Posts: 1,202
Well, I've taken the advice of Ivo and others on this thread and starting doing the dcraw conversions manually, using the 'clean as possible' parameters. The only drama I can see with this is that it converts my 14MB RAWs into 82MB TIFFs! Lucky I have plenty of hard disk space...
Reply With Quote
  #57  
Old 22-05-2014, 08:42 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,603
I think the PI dcraw options are probably OK. Will dig a little deeper.

The next issue not mentioned in the original post is pixel rejection parameters for calibration frames. I mentioned elsewhere, that because DSLR RAW data is mangled, integration parameters should not vary between frame types - bias, darks and lights, perhaps...

However, I now think, that "no pixel rejection" is a better option, for bias and darks, though bias frames are 'more' linear than darks. What do others think?

Flats are a separate case in terms of combination, but I still wonder if pixel rejection is warranted.

Lights need a degree of pixel rejection because they have been subject to unfriendly light sources - satellites, visual observers green lasers (bless them) cosmic rays and so on.
Reply With Quote
  #58  
Old 23-05-2014, 08:42 AM
graham.hobart's Avatar
graham.hobart (Graham stevens)
DeepSkySlacker

graham.hobart is offline
 
Join Date: Nov 2008
Location: hobart, tasmania
Posts: 1,962
dslr data

two questions...1/ if I have my lights and darks and flats & bias frames, when I load them into DSS is there a tick box to tick/untick to allow just calibration of flats with the bias and to ignore the darks or does this do it automatically?
2/ Is there a download link recommended for dcraw? or am I missing something here

NB I tried some data stacking with no interpolation and creation of superpixel and it does seem to be better, but I need to add my flats to make it complete (light box on way)
Graham
addit
I tried to calibrate in AA5 and although it looked good it didn't seem to debayer the image- can someone let me know where the menu item is for debayer in astroart 5 for canon raw files.

Last edited by graham.hobart; 23-05-2014 at 08:44 AM. Reason: addit
Reply With Quote
  #59  
Old 23-05-2014, 12:27 PM
rcheshire's Avatar
rcheshire (Rowland)
Registered User

rcheshire is offline
 
Join Date: Apr 2010
Location: Geelong
Posts: 2,603
Graham. 1. I don't know. You might have to do your flats separately. 2. Dave Coffin doesn't do executables, but they are available here for Windows and MAC.
Reply With Quote
  #60  
Old 24-05-2014, 03:13 AM
MrB's Avatar
MrB (Simon)
Old Man Yells at Cloud

MrB is offline
 
Join Date: Jan 2007
Location: Rockingham WA
Posts: 3,408
Quote:
Originally Posted by graham.hobart View Post
can someone let me know where the menu item is for debayer in astroart 5 for canon raw files.
Hi Graham,
Navigate to File/Configure RAW
Then tick the 'DCRAW color synthesis on load' option, see pics below.
It is also possible to debayer with the 'CCD color synthesis' module in the options tab of the pre-processor, but the DCRAW method is easier and much quicker when stacking, it uses less memory too.

Note that the image will usually look like crap when it pops up after stacking. I don't know why, but AA clips the image badly. Open the histogram and fix the end points.
Attached Thumbnails
Click for full-size image (AA-Raw1.jpg)
51.2 KB17 views
Click for full-size image (AA-Raw2.jpg)
30.7 KB16 views
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 04:15 AM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Advertisement
Meade Australia
Advertisement
Bintel
Advertisement
SkyWatcher Australia
Advertisement
Lunatico Astronomical
Advertisement
Celestron Australia
Advertisement
NexDome Observatories
Advertisement
OzScopes Authorised Dealer
Advertisement
Astronomy and Electronics Centre
Advertisement