Noise!!!!!!!!!!!

[raymo]

As my early efforts have all been short subs, generally up to 30secs, I
have been able to control noise using my very limited processing
ability. Now I have better PA I am using subs of 60, 90, and even 120secs.The noise I am seeing at those sub lengths is absolutely
horrendous, especially considering that the ambient has fallen to
around 12-14 degrees, and I am using 400 and 800 ISO. I haven't
got one useable sub over 60 secs. I took a single wide angle sub
of the LMC last week with my 18-55 lens of 50 secs. It was appalling.
Unless some one can tell me what I am doing wrong, or how to get
some control of noise, I'm going to have to resort to huge numbers of
subs at 30secs or so. It"s made worse by the fact that when posters
talk about using noise reduction features, I have no idea what they are talking about. I can't find any tools related to noise in my version of
PS [PS7]. Neatimage seems to help a bit, if you don't overdo it.
HELP!!!!!
raymo

[pluto]

Are you taking multiple subs and stacking them?
That should fix a lot of the noise. Also stacking with dark frames helps heaps too.

As for removing noise during processing, the best by far I have found is the noise reduction in Lightroom.

[traveller]

Download Deep Sky Stacker (free) and use it to your hearts content...

[raymo]

Thanks Hugh and Bo. I already use DSS and stack up to 70 subs at a time. I use in camera noise reduction, so I am getting an equal number of darks, and hence don't get any hot pixels. Iwill have a look at lightroom.
Thanks raymo

[alistairsam]

hi Ray,

do you know what the temperature of the sensor was when you took these subs apart from ambient temperature? Are you able to post one of the raw subs here?

I took 5 minute exposures last night at iso800 using my unmodded uncooled 1000d, wasn't too bad. sure there was noise, but not unmanageable. so there might be something wrong or it must've been a pretty warm night.

when you say noise, are you referring to the coloured specks or pixels all over the image? The in-camera noise reduction is normally quite effective. you mention that you can't see any hot pixels, but in the stacked image, you see a lot of noise? is this correct?

To effectively remove noise with darks, you need to dither the frames. not sure what software you're using for scope control or are you using the hand controller?
you usually enable dither in your main capture software that can talk to the mount and its usually done via the guider. the software will move the scope one or two pixels between every frame, but since you're not guiding, I'd suggest manually nudging the scope a tiny bit every few subs.

That way, DSS or other software will see the noise as a fixed pattern which hasn't moved and subtract it.
have you tried taking a dark and stretching that to see how much noise you have?

if you can't post a raw sub, can you take a screenshot of what you see when processing and post that?

Have you looked at software like BackyardEOS? makes capturing images very easy and its relatively inexpensive. It has a plugin for phd dither, but that's for when you start autoguiding.

Cheers
Alistair

[pluto]

I think you'll get better results if you disable in-camera dark frame subtraction and take dark frames to stack in DSS. That way each light frame will have the noise of all dark frames subtracted instead of only one dark frame per light frame.

[raymo]

Hi Alistair, I am not at home at present, and my images are on my laptop
there.I have recently bought BackyardEOS but haven't had time to get into it; hopefully next week, weather permitting. Because my PA is not perfect I can see slight movement of the target from one sub to the next
when viewing them on the camera's screen, so I thought that would take care of dithering. I have no idea how one would know the sensor temp.
The ambient recently has been 12-14 degrees. I don't use raw at all, just
stick to JPEGs [I know raw is better]; I'm just so sick of constantly
having to find ways of changing file types to suit software needs. Even
video files from my Canon are not recognised.
Just seen your post Pluto; I will try that, thankyou.
The noise I see is usually in the form of horizontal lines of fluffy knitting,
[that's the best description I can think of]. Sometimes it is like tiny
clouds of a combination of grey and pink. Another member recently
posted that he had the same horizontal lines in his images using his
1100D, so I assume that it is normal. I think that my main problem is that as an oldie I have little empathy with computers, and am in
totally alien territory. As an example, I can take a dark, as you suggest,
but as to how to stretch it to see how much noise I have, I wouldn't
have a clue. As fast as I learn how to do something, the next something turns up. You may usually get better results than film gave, but my
goodness you have to work for it. This is my last attempt to get it all
together. It is stupid and pointless receiving all this wonderful help
and advice, and wasting people's time, if it is just too much for me.
Thank you all so much.
raymo

[rustigsmed]

Hi Raymo, i think you are already using DSS to stack your images?

The easiest thing to do would be to take some darks. Just put the lens cap on or the scope dust cover and shoot about 10 - 20 shots with the same exposure length and iso that you have been using.

Thats all you need to do. When going into DSS, you have to select your light frames as usual, the only extra step is to select your dark frames too. Then let the program do its thing. It will automatically make a 'master dark' which you can use for a few months (or simply select the frames again for the next target you use).

cheers

Rusty

[tilbrook@rbe.ne]

Hi Raymo,

Turn off live view when imaging.
With live view on the censor runs at nearly 40c on a night that's around 17c.
This will also get rid of the lines in the image.

Cheers,

Justin.

[pluto]

You should definitely shoot raws, DSS will stack Canon raws so you you won't need to deal with converting them.

[raymo]

Thankyou Rusty, Pluto, and Justin. Justin, to clarify what you have said;
I gather you mean stop the camera displaying the image after taking the sub. The only time I use the actual Liveview feature is for initial focusing with my Bahtinov mask. The image is only displayed for about half a second or so before I take the next sub. I have now realised that a sensor
cooling period should be allowed between subs, maybe 10 secs or so.
Pluto, I am aware that DSS is happy with raws, but my comp won't open
them when they are put in a file, so I still end up having to convert them. If I remember rightly, I don't think my PS accepts them either.
Does simply having Liveview selected cause any heat increase, or does the screen have to be actually illuminated?
raymo

[traveller]

Backyard eos has a focusing function which I use over the live view function as it is more sensitive in my view.
Re taking darks, make sure you select the dark frame selection in backyard eos so the program is not thinking it is a light frame.
I also cool the camera in the fridge and wrap the camera in a plastic bag and put ice packs over the camera when aquiring dark frames to keep them around 5-7c. Not sure if it's the right thing to do but sure cuts down on the noise.
Bo

[raymo]

Thanks Bo.I had noticed the focus function in my look through Backyard, and planned to use it. I had thought about the fridge, but had read posts saying that the camera mirror fogs up. I also thought that the benefit
might be limited as the camera would warm up significantly over an
imaging session, but I suppose you could do several short sessions rather than one long one, and put it in the fridge between sessions.
raymo

[pluto]

The dark frame selection in BYE just changes the file name, AFAIK it doesn't change anything else.

You want your dark frames to be at the same temperature as your light frames. If you are taking your darks at a lower temperature than your lights then they will have less/different noise and you won't get as much benefit when stacking. If the temp is drastically different then it may make your stack worse than if you didn't use them at all.

[raymo]

At least by using the in camera noise reduction I know the temp will be the same.
raymo

[Oggie]

I think this can be fixed by installing the free Canon Raw Decoder from Canon's site: http://support-au.canon.com.au/contents/AU/EN/0200255003.html

[raymo]

Thanks Peter, I'll do that.
raymo

[jsmoraes]

Up to now I only work with DSLR Canon. Model 350D. My sessions have around 30 - 50 minutes. Depend of which battery I am using.

My experience says:

1) dark frame: I use only 5 or 10 as normal photo. Same ISO and exposure. It is captured after de session. They will used in DSS as dark frame.

2) The native ISO of 350D is 800. I work with this ISO or less: 400 and 200. ISO 1600 the noise is absurd.

3) the time of exposure is 240 sec maximum (4 minutes). Because of light pollution,the glow of background is very high if I use more time. And they are sequencial: there are no time between them.

4) if the seeing is bad, I perceive that the noise is more strong. Please don't ask me why. As I told ... I perceive.

5) I don't use the internal Canon noise filter. I don't see high improvement on the result.

6) I never perceived difference in intensity of noise of first frame and the last frame.

6) After stacking with DSS, I don't make any adjustment. I use the original Autosave.tif - 32 bits. In Photoshop I convert it to 16 bits, using the Local Adaptation option, and make adjustments with the HDR curve. Radius: 1 or 2 px and Threshold 0,50 or less.
This conversion can increase the noise. You must experiment to get the better image: details versus noise.

7) to reduce the noise, I use in photoshop noise reduction in each color channel. My camera has more noise in red. The minor noise is in green.
There are no trick to it. Some times I create mask of background. Some times not. Each case is a different case.

8) Some times to the final work I use Filter blur -> surface blur (radius = 2 or 4 and threshold never more than 4) , noise -> dust & scratches (raise = 1 threshold = 1 and opacity layer less than 50 %)

You can check my photos in astronomia.pousadavilatur.com.br. My great problem is focus, because 350d hasn't live view.

[traveller]

Useful info. Maybe I should cool the camera before taking light frames to match....
Bo

[LightningNZ]

My setup is quite unsophisticated (because I am essentially lazy) but I can control the noise from Canon 300D reasonably well if the ambient heat isn't too high.

Here's my general observing session:
1) Take a bunch of light images of between 30 seconds and 2 minutes depending on much effort I've put into polar alignment.
2) Every hour I take 5 or 6 dark frames, so I can at least get the temperatures matched to the light frames. You don't need many dark frames with modern cameras and the 300D is likely the oldest camera you'll see on these forums - it's very badly affected by high temperatures.
3) Once my tiny memory card is full (about 70 images) I take the scope (AT65-EDQ) inside with the camera still attached and pull down all the images to my computer.
4) I take flats by photographing my LCD screen (set to white) at close range. I take maybe 20-30 flats and move the scope/camera setup around as I do so to even out any imperfections in the LCD panel.
5) Run it all through DSS, saving the resulting file as TIF (no file compression, no losses).
6) Post-process in Photoshop.

I think you'll find that if you turn off the built-in noise reduction you'll be able to get a lot more starlight for time outside and the few darks you get will give a smoother result than a single dark per frame.

Hope this helps,
Cam

[raymo]

Thankyou all; there's stacks of stuff there to keep me experimenting for months, and it will be months because we've moved into the time of year when suitable nights are about 1 in 10 if I'm lucky, [but that's another
story].
raymo

[lazjen]

I'll add to the chorus of not using in camera noise reduction.

I build a library of darks for each ISO/duration combination. They're named with the temperature details, so they're easy to match with the lights (also have temperature details in the file name).

Liveview is off (it's power hungy and generates a huge amount of heat), focusing, etc is all done via the laptop (BackyardEOS mostly, sometimes APT). Images are stored on the laptop, not the camera.

I've got a power adapter for the DSLR that I run through my break out box (so it works for 240/12V) - this also means I'm not limited by battery time and can go all night.

I almost always use PHD dithering when taking images, so there's a very slight position difference - this improves the result in DSS (which, btw, you should be using the latest beta if you aren't already).

Flats/Dark Flats are taken each session (about 10 each). For the backyard I have a convenient house floodflight near by I can switch on that gives a nice glow to the white cloth over the scope. When I'm at a dark site, I either do the flats in the late afternoon or use the car lights. :)

[raymo]

Thank you all sooo much for the wealth of information. I have printed out
all 7 pages of this thread so I can peruse it at my leisure, and make some changes to my method of image acquisition.
raymo

[raymo]

Perhaps Justin or Chris could clarify what they mean by having Liveview
switched off while imaging[Justin said that it would get rid of the lines
on my images]. I only use it for maybe 20 secs or so with my Bahtinov
mask to get the focus right before starting the session. I will now be focusing using BYE. I don't understand why I would want Liveview on
during imaging.I like to store my images in the camera as well as on the laptop as a backup. This doesn't cause noise does it? Sorry, I'm not very
tech savvy.
raymo

[alistairsam]

hi Raymo,

you mentioned that not even one of your 60 sec subs was usable, are you able to post that here, or even a screenshot of it? would be good to see the noise you're referring to.

focusing with software is much easier.

You also mentioned you save as jpg's?

My suggestion would be to change the settings to save as raw and jpg, use the jpg's to browse and select the ones you want to discard, and then use DSS to stack the respective RAW's. you'll find that results will be better this way rather than stacking jpg's.

Cheers
Alistair

[pluto]

There is no reason to leave Liveview running while imaging but some people might do it anyway and this would generate excess heat and battery drain, you're obviously not one of them so you don't need to worry :D

Writing images to the card would generate a bit of heat if you were taking shots quickly and it was constantly writing but writing an image every 30/60/90 seconds isn't going to do anything negative IMO.

In terms of heat, using Liveview on the camera and focusing using BYE would be the same, as the mirror is up and the camera is reading and processing image data, but I wouldn't worry too much as 20 seconds to focus is negligible compared to a night of imaging.

[raymo]

Hi again Alistair, thankyou for your continuing interest in my problems.
As I said the other day, I'm not at home at present, and don't have
access to my images which are on my laptop there. Even if I was there I wouldn't know how to send a screen shot. Another way of describing the
noise would be to imagine horizontal rows of seismic readouts of earthquakes, but with small spikes like little tremors would produce.
Another 1100D owner recently posted about the same pattern he was getting. The lines are really close together like knitting would be.
Research is a little difficult for me, as my laptop's memory is nearly full,
so I discard pretty much all of my substandard material. I just thought of another suitable description: barbed wire with the barbs almost touching each other. Even if I don't have any individual subs showing this effect, I will presumably have some finished images showing it. I usually minimise
it by saturating the black, but that loses me other detail of course.
raymo

[raymo]

Thanks very much Hugh, that cleared that up.
raymo

[nebulosity.]

I have the whole screen turned off when I'm imaging with the 1100d, not just live view. I think it's the screen being lit that corses a lot of the warmth, not just the live view.

My sequence when imaging is this: turn on live view, focus, press live view button again to turn off live view and then press 'display' button to make the screen go completely black.

Imaging like this I get hardly any noise banding ( I think that's what they call what your talking about) previewing the image for a few second after its taken it doesn't corse any dramas.

Sorry if this is already what you do, but it seems to work for me.

Cheers
Jo

[alistairsam]

hi Raymo,

are the lines anything like this?
http://i96.photobucket.com/albums/l199/UKAndy/20110325HorizontalBanding.jpg

is it vertical or horizontal?

There seem to be a number of reports of vertical and horizontal banding with the 1100d.
I've never seen it with my 1000d even with 10min exposures at 20deg C, sure lot of noise, but not bands.

I would've mentioned changing USB leads but you said you save it to the card...
when you do use USB to capture, make sure it is good quality and isn't running too close to your dew heater supply cable or other power leads.

See if you can capture BIAS frames. there's a long thread in SGL where they've got some good results with just using flats and BIAS, not even darks, to remove the banding.

make sure you save as raw+jpg when using BackyardEOS and change the image type to Light/dark/bias so the files will be tagged/named accordingly.

If your stars aren't elongated, you shouldn't be discarding frames.

Cheers
Alistair

[raymo]

Yes, that is what I do Jo and it certainly doesn't work for me. Up to now I have been imaging manually[which is a drag when getting a lot of subs],
so the screen only lights up after each shot for a fraction of a second
before I take the next sub. I am going to turn the screen off altogether
after focusing, and see if that helps, and also leave 10secs between subs
for sensor cool down. Oh well, keep trying.
raymo

[paul71]

Hi raymo, I see you are still having problems with horizontal banding,
my 1100d is still giving me the same banding issues as you are having.
There are a lot of reports on the net relating to the 1100d and horizontal banding,when I had the problem last my sensor temp was in the high 30s,eg 38-39 and I was taking 5min subs.so maybe this was a big contributing factor.
I think when I am imaging again next all I can do is try a lower iso around 400 instead of 1600 and maybe shorter subs.
The thing that disappoints me most is that I bought the camera new and then sent it to the usa to hap griffin to have it fully modded, if i had known that the 1100ds were prone to this sort of banding i would not have wasted my time and money.
Cheers Paul.

[raymo]

Hi Paul, How do you know what the sensor temp is? I'm using lower ISO as my PA improves, allowing longer exposures, but have found that the noise at 1600 is hardly any less than 6400. Next step 400 and 800.
I don't frequent any other forums, so wasn't aware that this banding was
being discussed elsewhere; certainly none of the respondents to this thread have mentioned knowing of it. I wonder if it is confined to the 1100D. Assuming that this is an artefact of this model only, I certainly wouldn't have bought one either. Perhaps I was fortunate that I couldn't
afford to get it modded.Perhaps we could PM each other if we see anything of interest or importance about this problem crop up on the net.
Cheers raymo

[alistairsam]

hi Paul, Raymo,

I'd strongly suggest you'll acquire BIAS, Flats and Darks and then stack your images.
From reading forums, this seems to remove the banding and Justin here who has some very nice results from his 1100d from memory also uses flats, darks, bias. I wonder if he's noticed any banding.

I believe this has been noticed with 450d's, 300d's and 350d's, but not as much as the 1100d. I'm surprised as the 1100d is known to have low noise.

Cheers
Alistair

[raymo]

Hi Alistair, thanks for your contribution; I'll digest it along with all the other info I have printed out. I am still trying to find out how to know what the sensor temp is.
cheers raymo

[scagman]

Hi Raymo,

The sensor temp, along with a heap of other data is recorded/embeded in the image. If you right click on 1 of your images, select "properties". This will open a window, click on details and should be able to find it in there. You can also use firefox with the Exif Veiwer addon to view it as well.

Hope this helps

Cheers

PS here's a link with some info on Exif files http://photographylife.com/what-is-exif-data

[raymo]

Thankyou so much John.
raymo

[raymo]

Been through 6 photos; no mention of the sensor amongst the scores of
items. The camera section has no info at all. I have looked at older images
taken with a different camera, and all the shooting details are there along
with the camera make and model, but still no mention of the sensor.
raymo

[rcheshire]

To be honest Ray, sensor temp is nice to know, but if you stick to the calibration formula and religiously take darks and flats immediately after or during imaging (bias if your software needs them), you shouldn't have a problem. Your image will be easier to analize for problems after proper calibration.

Temperature differences are inevitable with uncooled DSLRs. What matters is that you are taking calibration frames around the same ambient air temperature. If you take enough, the average is better and accounts for fluctuations in sensor temperature.

If you take your darks in an identical sequence to your lights the frames should be more consistent. It's all about consistency when working with temperature. That's why regulated cooling is an advantage.

Dithering, and I don't mean by a half or 3 pixels, but aggressively, for uncooled DSLRs, (12 - 15 pixels) will take care of a host of noise problems as well as inevitable calibration artifacts, due to slight temperature mismatches 5C or so.

An uncooled DSLR sensor is not at all like a cooled CCD.

Rant over...

[raymo]

Hi Rowland, I'm not tech savvy enough to be wanting to know the sensor
temp for practical purposes; I keep having people talking about it, so I was curious to know how to find out what sort of temps mine runs at.
The sensor temp doesn't seem to be where John said I should find it.
Searching through my images I find that some images have the imaging
details[camera make/model, shutter speed, etc : and others have a blank
section where the camera info should be. Mysterious. This digital business
is like a twisting road, you've no idea what's coming next.[except clouds].My next adventure is to get my BYE in use. I believe I can dither with that, but need a plug in.
raymo

[rcheshire]

I know... complex...

Keep it simple. Time and air temperature... consistency is the key.

[scagman]

Hi Raymo,

Sorry for the bum steer, I thought it would be in exif details, I use a ccd not a dslr, which uses the fits file format and has a fits header similar to the exif header, so assumed it would also be in the exif.

Cheers

[raymo]

No worries John, I expect somebody will eventually tell me where to find
it, assuming that you can with my camera; it's only a budget level one
after all.
raymo

[tilbrook@rbe.ne]

Hi Raymo,

Are you imaging via laptop or on camera?
If by camera, live view will usually shut down after a while unless you have adjusted the setting in the camera.
If via lap top, the live view won't shut down unless you turn it off.
I've forgotten quite a few times only to find a very noisy sub at the end of the exposure.
Jo is right about the display, turn that off as well.

Cheers,

Justin.

[raymo]

Hi again Justin, Up to now I have been old fashioned and stood at the camera and done all the exposures manually. I turn Liveview on, focus
with the Bahtinov mask, turn Liveview off, and as the screen lights up at the end of each exposure I immediately take the next exposure, so the screen lights up for less than a second at a time.The only time I use the laptop is when using my webcams.
raymo

[raymo]

Alistair, I came upon this one I was playing with. Disregard the colours.
The noise pattern I mentioned shows up better with less colourful images, but is still obvious. This was 50x 20secs @ ISO 400. I'll have a look for some longer individual subs.
raymo

[rcheshire]

Ray. That is either horizontal banding, fixed pattern noise and just a function of your sensor operation or the result of stacking low SNR images without dithering (or both) - particularly DSLR images. Explanation is complex, so suffice to say that dithering will most likely fix this. If you can move your mount in RA or DEC between images with your hand controller without the image going outside FOV and repeat between images - random is OK - I think you will see an improvement.

[raymo]

Thanks Rowland, I'll give it a go.
raymo

[LightningNZ]

I get this too Ray with my Canon 300D. You see less of this with a less noisy camera but as Rowland says, dithering will sort it out. I've got to get dithering going myself.
-Cam

[raymo]

Hi Cam, Let's not dither about, and get on with it then. People on this thread seem surprised that the 1100D has this problem as it is thought
to be a low noise camera.
raymo

[doppler]

Hi guys , I have noticed this "aberation" in both my 1100d and 450d. It seems to be more noticable when the sky has more background glow. (ie moon or streetlights) I wonder if it could be a cannon software glitch?

Rick

[raymo]

Hi Rick, Yes it does seem to be more noticeable under certain conditions, but I don't think that it actually varies. I think it's always there if you look
hard enough.
raymo

[rcheshire]

The consumer CMOS sensors installed in todays digital cameras are active devices. Each pixel is composed of a photosite (the light sensitive bit) co-located with a processing circuit. Consequently, CMOS sensors have characteristic patterns of electronic noise.

This is normal, but typically different for each camera model. Banding is a function of CMOS operation. Some worse than others. To obtain good results from CMOS images, they must be calibrated. In many cases dark and flat frames are adequate for this purpose.

There is nothing wrong with the camera, it's just that we are asking it to do something for which it is not specifically designed - unlike CCDs which are passive devices (please correct me if I am wrong about this) and usually cooled.

I would use in-camera reduction if taking only a few images at a time. It is very convenient, though not necessarily suited to large image sets.

This digital low light imaging thing is hard work at times and there are no shortcuts.

You are stacking lines of electronic noise layer upon layer - the effects are devastating to your data.

[raymo]

Thanks once again Rowland; I'm doing my best to digest it all.
raymo