Desktop computer advice needed

[gregbradley]

I am wanting to upgrade my computer for image processing.

I am currently using a HP Pavilion Laptop i7 with 8gm RAM and 1 TB HD.

Its way too slow and too little RAM.

My cameras both do around 32mb for a single image. So if I take multiple hours of images I may have as many as 30 x 32mb image files to process at one time.

I use CCDstack to do the initial processing and its arduous and slow.
It usually will limit me to about 9 x 32mb images before it starts to reject them. Doing some CCDstack operations can take quite a while.

This makes image processing a bit of a drain.

I am thinking of:

1. A desktop not a laptop.
2. Not a HP. This laptop is starting to fail and its less than 2 years old.
3. Latest 6th gen i7 4ghz processor. I have read some refer to Skylake and it seems to be the latest.
4. 32gb DDR4 RAW minimum perhaps 64GB. Here is where I am not sure. Some computers mention quad channel for DDR4 memory allowing 8 slots and others mention dual channel allowing 4 DDR4 slots. I assume as RAM is critical that quad channel is the way to go
5. 3 TB hard disk.
6. 500gb SSD.
7. Upgraded cooling and power supplies.
8. I am not sure how important the graphics card is. Geforce seem to be dominant. Should I go for an upgrade here or is it not important? I won't be using the computer for gaming or if I do its very minor usage. However doing image processing does require accurate colour so perhaps a good card makes sense?
9. I also read one supplier talking about a hyper speed tech where programs open and close instantly. That sounds good. Is this generally available?

Anything else I should consider?

Greg.

[alocky]

Hi Greg - it's not always hardware related, but I'd definitely go for a desktop, two monitors, and go for the latest multicore server setup (these become obsolete overnight so I won't suggest whatever weird name and number they've got today), make sure you've got as much ram as you can shove in it, at least two ssd drives and get a decent graphics card, because a lot of software like photoshop can actually use the gpu instead.
Then use pixinsight for your calibration and stacking, and use the ssd drives for throughput.
I've been processing 100s of raw files at a time from my full frame Nikons for 5 years on my current setup and it's still quick.
You might want to consider an external NAS device for long term archiving.
Anyway - not a laptop. And not really a gaming machine either.
I got mine built to the specs we use in our industry for managing and visualising truly enormous geophysical datasets, but went through a local small company who were so excited they wanted an extra few days to 'benchmark' with their favourite games...
Cheers
Andrew

[RickS]

Greg,

1. A desktop system is a good choice for performance
2. Don't have a view on particular brands. Have been burnt by most of them at some time :)
3. Skylake is the current latest. Early next year Kaby Lake desktop CPUs will be available but will only be an incremental improvement
4. Lots of memory is good. I have 24GB in my 6 year old desktop system and it is perfectly adequate for very big integrations in PI. Sounds like CCDStack isn't as efficient. Quad channel DDR4 will give max performance with 4 slots filled (one channel each.) 8 slots is good if you decide you need more RAM later.
5. Lots of HDD makes sense
6. SSD for your boot/Windows disk will certainly improve boot time and application start times.
7. Some power & cooling headroom is useful
8. You don't need a fancy gaming card. Have a look at the Nvidia Quadro range of desktop workstation cards (you don't need a high end one.)
9. No idea what this is. SSD makes a big difference to program start times. For processing you won't care that much since you'll be running the same program for hours, or days :lol:

Cheers,
Rick.

[RickS]

That's certainly true if you're doing a lot of PS. I live in PI the whole time and it doesn't use the GPU. Neither does CCDStack.

As Andrew says, PI is very efficient at calibration and stacking and uses multiple cores/threads to great advantage.

[pluto]

Even if you aren't gaming I would still go for a decent entry level gaming card, like a GTX970 or GTX1060. The reason is that most Adobe software now relies on GPU for many functions and this will continue to expand with more and more functions being shifted to the GPU.

Definitely get an SSD, and if your case will fit it you can get some extra cheap fast storage by buying 3 or 4 1-2tb drives and raiding them. I do this at work as a kind of scratch drive and it's great. My OS and applications still run from an SSD though.

[pluto]

Not sure what you mean by this but a Quadro is way more overkill than a gaming card, and much less bang-for-buck for the kind of stuff Greg will be doing.
I've used many Quadros in the past and have never seen the value of them over a good gaming card - and I work in CG using the kind of tools those cards are designed for.

[RickS]

If his bottleneck is doing large stacking runs in CCDStack then a gaming card won't help at all. It will only make sense if he's doing hefty processing in Photoshop. An entry level, but good quality, workstation graphics card is a reasonable alternative to consider. Astro image processing is not very graphics intensive.

[pluto]

I agree that a gaming card won't help for stacking, neither will a Quadro. However, I think Greg does do all his other processing in PS...?
It's by no means the most important component in his build but I wouldn't build a box, for processing in PS, without at least an entry level video card.

I guess you're right that a workstation card could be considered, but personally I wouldn't recommend a Quadro over a GTX for use with Adobe software as, for the money, you're not going to get any performance benefit. Plus, in my experience, because you're using different drivers you can have problems running certain software which would run fine on a GTX. That's just my experience though, and it's why I don't buy Quadros anymore.

[gregbradley]

Thanks for the advice guys. Photoshop is not really slow even on my laptop. It takes a little while to boot up but I couldn't say its slow.
Its CCDstack that is slow. Callibration and forming master LRGB files takes the time.

To do a hot pixel data reject on 10 32mb files in CCDstack takes a few minutes.

With 8gb RAM I can only open 10 files.

I'll have to apply what I learnt at your PI workshop and get the calibration workflow going in PI. It sounded like once its setup its quite fast and you can reuse all the setup.

Greg.

[pluto]

I recently stacked about 30 raws from a 5DmkIII in PI on my little MBA11 with only 4gb of RAM (running Win10) and it worked fine. Took a couple of hours though...

[ChrisV]

How do you feel about DIY ? I got sick of prebuilt nonsense and have done a few home/work desktop builds getting components from the likes of MSY. Can do it way cheaper and get exactly what you want. Whack the operating system etc on a smallish 250-500GB SSD and get a few mega-hard drives. And don't go crazy on a graphics card.

The most expensive part is usually WINDOWS ! And the most complicated = installing it.

[gregbradley]

CCDStack has never really handled memory very well and has been a very memory intensive program. Hopefully 64gb ddr4 RAM will suffice.

I would rather someone with more current knowledge of compatible bits and pieces to put something together rather than DIY. But its a good idea.

Greg.

[Somnium]

i run ccd stack with 16 gig of ddr4 ram and it seems to run fine. granted my image size is probably not nearly as bit as yours. i am running an 8300 chip.

[Windston]

I also really, REALLY recommended ditching prebuilt pc's. I am 16 and have build over 15 pc's in my years and haven't had a problem with one. As long as you dont throw the parts, you will be fine. It is a fool proof system, if the plug doesnt fit, it isnt meant to go there! My first one took me a good 4 hours, now I can do it in under 45.

You can buy new or used (Overclockers Australia is good but you have to be a member for 3 months, but they do have really really cheap used pc parts classifieds, never had a problem there either), or new, from Umart, MSY, Mwave etc.

You could do a NAS if you wanted and feel that you would benefit from it, I just installed one at my house, moving the file server from my pc to a dedicated server pc in the cupboard with all of the file storage for the family on there, it is really great and quick especially if you have internet. I can come in from shooting and transfer all files within one minute, or even over wifi when I am near enough to the house. Otherwise just make a shared folder over the network for you desktop so that you can transfer, otherwise just do it oldschool with USB hard drives. From there I use my custom built desktop to process.

I have an i7 4770k which is a great chip, really quick, but only have 8gb of ram at the moment which is a killer. Star Tools uses it all and often crashes my pc! :lol: Over 16gb is a must, over 32gb is a tad overkill.

Anyway it is much much cheaper, customization and easily up-gradable compared to prebuilt if you are willing to build it. Having a friend on you first build will often make it easier, but I did it at 12 after a few youtube videos without and problems so I am sure that you will be fine! :thumbsup:

Cheers
Dan

[gary]

Hi Greg,

I concur with those who suggested you should build the machine yourself.

It is not difficult and can be done in an afternoon.

Suggest you start by looking for the best quality motherboard with the
specifications you require. For example, most at the moment are built
around the Intel Z170 Chipset.

The Gigabyte Ultra Durable range might be a good starting point to look -
http://www.gigabyte.com/mb/100-ud/Model
http://www.gigabyte.com/mb/100-ud/Durable-Safe

Then pick a CPU with the right price/performance point to match the motherboard socket.
At any one point in time, there will be a sweet spot which you can
ascertain by looking at the PC suppliers price lists.

32GB of DRAM.
SSD, hard drives, etc.
Consider two identical hard drives mirrored in a RAID 1 array for redundant
reliability.

Then pick a good case. High quality with slide-out disk storage bays and
plenty of quiet fans and removable filters.

Add a power supply.

You can always add a graphics card with GPUs later if need be.

The motherboards all come with manuals that have diagrams of the various connectors.
Once the motherboard is screwed into the case using the standoffs, the cpu goes
in easy enough (it comes with its own install instructions) and the DRAM just snaps
into the sockets. Which cable plugs into where is the only task that requires any
concentration and most of the time the plug types and sockets are unique enough that
it is difficult to get any swapped. The octopus of cables from the power supply has a multitude
plugs but it will become self evident which plug into the motherboard and which power the SATA drives.

In the end you will end up with a system with the best components (you chose them) and the best bang
for your buck and one that you know was assembled with care.

[lazjen]

And once you start building your own machine, ditch Windows and build the OS using Linux. ;)

Building your own machine is the way to go. It's not that difficult and you get far better value for your dollar - and exactly what you want. I've been building my desktops for years now and each system has been more powerful and quieter than the previous one.

[sil]

I don't think the OP will be able to build a PC, if they could the original questions should be part of their knowledge.

I can highly recommend getting a machine built from Scorptek, specced to what you need: fast cpu, more ram, storage, gfx card etc. the build damn good machines and have a variety of specced machines on their website for all budget needs.

I used to build custom machines to compete in the CG industry when SGI started to get non cost effective.
I recommend Intel CPU over AMD as software is typically more stable and optimised to make full use of cpu features than amd ones.
Also I prefer Radeon gfx cards, over Nvidia for driver stability issues. However while its true some software can use GPUs for processing, they only use either nvidia OR radeon as each has its own methods of GPU processing which are incompatible with each other. So check the software you use to see if it uses nvidia or radeon gpus and decide based on what your actual software uses. a good monitor and colour calibration is important too.

Just speak to Scorptek about a machine for your needs, theres some real inappropriate advice in this thread.

[rally]

Greg,

Is a 27" iMac out of the question ?
Or a Mac server with multiple 4k screen support !

The iMac has a colour calibrated screen to die for, multiple big screen support
OSX is Unix/Linix based and so handles multicores and multithreading very well
PI is supported natively as is Photoshop etc

Anything that is Win specific you choose to keep can be run in a virtual machine and you can mix and match Win and Mac programs using Parallels.
In fact you can have multiple Win OS's running different programs simultaneously and creating a new virtual machine for say testing new software that you may or may not want to keep can be as simple as a single file copy.

Best of all worlds really

[pluto]

The iMacs do have nice screens, and they are very shiny, but only i5 processor? and the price!!! :eyepop:

I didn't think Apple made servers anymore...? We have an old Mac Pro shoved next to the rack as a file server because they stopped making them...

[rally]

Not sure where you are looking Hugh?

My first iMac 6 years ago was an i7 and they are still available in various i5 and i7 configurations today.

You build it online and get it how you want from Apple - you can get an i7 in a number of processor options.

I just fiddled online and configured a 4.0GHz quad-core Intel Core i7, Turbo Boost up to 4.2GHz with 32gb RAM, AMD Radeon R9 M395X with 4GB, 1TB flash drive
The big flash drive made it a bit exy, but a standard 3TB hard drive was OK.

The "servers" (traditional box without screen) are called Mac Pro's
The standard processor in them is a 3.5GHz 6-Core Intel Xeon E5 processor but if you have plenty of $$ you can get a 12 core Xeon
They come standard with Dual AMD FirePro D500 with 3GB VRAM graphic cards.
Expensive as a personal PC but powerful and definitely still available !

You dont get the variety of options if you buy the bits and build a WinPC yourself, but you also dont get the potential problems either.

[pluto]

Ah, I was looking on the Apple site and I saw you could customise RAM but didn't see anything about processor, my bad :)

I use Mac Pros all the time but I wouldn't call them servers, they're a bit pricey for something that lives in a cold room out of sight. They're also significantly more expensive than an equivalent non Mac with the same bits in it, albeit in a much more attractive case.

And my experience doesn't really agree with your idea that you'll have less problems running OSX over Windows. We have far more issues with the Macs at work than the Windows or Linux boxes, and that's in the Mac's native habitat (Design and film production).
I'm typing this on my Mac Book Air 11 on which I'm running Windows 10 and I can say with certainty that it runs so much smoother and more stable in Windows than it did in OSX, plus I don't need to virtualise anything to run the programs I want. Kudos to Apple for writing decent drivers for Windows for their hardware.

However, I'm not trying to convince you or anyone else about what to spend your money on, or your time with, whatever works for you :)

[sil]

Are there any astro programs for mac that are awesomly better than on windows or linux? I recent inherited a mac mini and wondering what to do with it, if anything. Think its 2.0 i7 with 2GB RAM and 320GB storage. don't want to turn it into a digital photoframe, I dont use the ones I have as it is. Was thinking finding a touch screen and making it a stand alone mandelbrot explorer kiosk.

[pluto]

You can run PixInsight, it's awesome on every OS :)

To answer your question though, I'm not aware of any astro related application that is OSX only, and which is better than an alternative available also on other OSs.

[Camelopardalis]

Oh I love sweeping comments such as that, especially from one who's just demonstrated they can't use an internet browser properly ;) :P

In MY experience of working in multi-platform environments for more years than I care to remember, each platform has their own foibles that need to be accommodated if they're expected to play nice with others. Two experiences, different outcomes...take your pick :lol:

[Camelopardalis]

Nothing better or worse than the other options. EQMac does the same as EQmod. Stellarium, PHD, PixInsight are firm favourites on any platform. You can use Wine to run Windows programs, and most Linux programs distribute the source code that you can compile on a Mac. It's a middle ground, but nothing sticks out for astro :shrug:

[pluto]

Yeah I never quite worked out these new fandangled browser thingies ;)



I totally agree with this. That's kind of the point I was trying to make (perhaps poorly) - in my opinion there's no perfect system and statements to the effect of "run a Mac, you won't have any problems like Windows computers have"* are inaccurate.

*I'm paraphrasing of course

[Camelopardalis]

:lol: :lol:

Completely agree, there is no perfect system, it's more just a case of "whatever works" IMO to get folk to look past the technology and get on with their work/processing/artistry :D

[g__day]

If you want real speed - confirm whether your processing software can either use multiple cores or Direct Compute.

If the answer to multiple cores is yes - well the 10 core I7-6950X isn't cheap but will be unbeatable. I would go 64 GB of RAM and add two of the latest PCI Express SSDs - costly but with 3GB/sec throughput vs 500 MB/sec of a SATA3 SSD - that is six times the speed.

If your software can use Direct Compute - well a uber powerful NVidia GTX 1080 gaming video card makes huge sense - having 2,560 processing cores will lift your imaging processing beyond belief (I have seen video of 4 GPU monsters doing real time x-ray tomography - using specialised software to handling the 10 TB/sec processing load across the 10,000 ALU cores).

I would expect most image processing software doesn't yet take significant advantage of the huge parallelism inherent in modern 3d accelerators - writting decent shader programs still isn't all that simple - give it 3 - 5 years.

I would add get a great - high quality power supply, case and motherboard - and the 30" Kogan monitor is a decent thing to initially pair it with!

You could probably build or buy a rig that for $2K - $3K could do most of what you want - but you could double or triple this spend easily depending on how much headroom you need in your processing capacity, how important your time is to you, your budget and where you need to spend $ to get real world returns.

[Camelopardalis]

Most astro software isn't multithreaded enough to make use of more than about 4 cores. I've profiled PixInsight while performing typical image processing tasks and it doesn't saturate all the cores on my i7 all the time. High throughput storage and lots of it is essential though.

I reckon certain modules would be a good case for it, it's largely SIMD after all...

[silv]

re the fast storage access:
the Thunderbolt 2 interface (Apple) for hard drives (and other periphery) gives an IO throughput of 20Gb/s .
(bit, not Byte)

PCIe goes up to 16Gb/s and USB 3.0 and SATA are way below that.

but... that access speed is not necessary for still image processing.
The software pulls the whole images into its workflow before processing anything (that's a guess, I didn't code them, but it sounds logical).
So accessible CPU and RAM, accessible for software and OS, are far more important for this task.
20Gb/s is good for video and music editing.

yeah, go into a good shop and describe what you need it for, like Sil suggested.

[Windston]

A PC of that caliper is way overkill for astrophotography software! :lol: Anything over 32GB RAM and I dont think it will be used, and most software wont even make use of the 10 cores, however that is a nice CPU, and if it was to be a purely AP processing system, and not used for 'gaming', then a 1080 would be a waste, better just to get either a good CPU or good GPU, and I would put money on the CPU every day of the week. Also I think the 2,560 processors is CUDA cores or Stream predecessors, and not individual cores like a CPU, and are often not much faster than CPU based rendering in the limited software that it supports. And as for the PCI SSD's, sure, they are quick, but SATA3 is already very very fast, M.2 even more so, which would be a more suitable option, cheaper as well, a 500gb M.2 SSD with a few 2-4TB HDD's in the rig would work really nicely. I dont store any of my AP RAW data on SSD's as the space is simply to expensive for such little gain in processing, HDD's dont bottleneck the processing of the image at all. (I mean, it makes it only very very slightly slower, an SSD in this application is nothing to write home about.)

I have a i7 4770k currently not overclocked, but for certain pieces of software like Star Tools, I have noticed a huge difference from the i5 4590. 4-> 8 cores. That being said, I only have 8GB of ram at the moment and it just sucks, I had 24Gb recently, and it was awesome for memory hungry software, but I had to take it out! :(

But it really depends on his budget, but an expensive system isnt always a better one. Price to performance on a graph is not linear. Spending 300$ will just get you a crappy pc, $800-$1200 in my experience is the best bang for your buck, anything over $2k, I rarely find people with reasons to justify spending over double for a minor performance upgrade. I have built PC's for kids for $500 that can run games like battlefield on high, and $2.5k just to get to ultra! :lol:

With websites like OCAU (An ozzy Pc forum) and their for sale classifieds, it is possible to build a really powerful PC used, from gear that is only about 1-2 years old for a fraction of the price. I have built 4 PC's in the last 6 months this way and have only once had a semi faulty product, which was refunded immediately. I find 40% cheaper is average. However you do need to be a member of the forum for 90 days in order to gain access.

Cheers
Dan

[pluto]

This is interesting.
I just ran a benchmark in PI on my workstation (2x10 core Xeon E5) and the most I saw it peak to was 86% total processor usage, and most of the time it was way down near 15% (100% would be all 20 cores saturated). I then ran an Image Integration on 10 subs and the most it peaked at was about 18%.
Then I ran a deconvolution, with default settings, and again it only peaked at about 20%.

I recon you're right that the bottleneck is unlikely to be the processor in a modern box and it's much more about really fast read/write speeds (my ssd is pretty old and only does around 300MB/s).



Thunderbolt 2 (and 3) is awesome but make sure you check what speed the drive can actually do.
For example we use a lot of the Lacie Thunderbolt2/usb3 rugged drives at work and on the box they quote the same speed for both. Unsurprising as the drive is just a standard 5400 rpm 2.5" which has a max speed of about 120MB/s - well below the potential speed of both Thunderbolt 2 and usb 3. (I keep telling them to save a $100 per drive and get the usb3 only version but no... Thunderbolt 2 is faster!)
Even an average SSD with a read speed of around 500-600MB/s is nowhere near the potential speed of Thunderbolt 2 - though having an external drive at those speeds is pretty awesome :)

[sil]

Pretty much what I expected, thanks. Never been a Mac person so was hoping there was something niche I haven't heard of. Mac mini is packed away in a box for now

[silv]

pity. can't you think of a local charity/preschool to donate it to? MacMini, probably 2011 model?, is such a reliable little machine and with those specs still good with OSX 10.11 (free OS upgrades) and as an office workstation. it would be sad to just let it collect dust.

[g__day]

The take away from my earlier post should be a really top quality motherboard and Power Supply is always a great investment; the rest of the gear should be selected to meet your needs.

However realise that coding queue based, multi-threaded software isn't all that hard once you understand the basics (my thesis in the early 80s was on coding frameworks for parallelism and language design...) - so 35 years on hopefully this practice will one day catch on once all the frameworks are set up so even semi skilled idiots can do it ;)... So maybe in a few years it will be really, really common - as multi cored machine parallelism isn't going away.

To really leverage GPUs you need problems that 1) don't need masive amounts of data transferred to a dedicated hardware unit for just a short period of time and 2) benefit from mass parallelism by being able to be broken down easily into a set of much smaller, individual problems to solve.

DirectCompute (Or OpenCL) is taking off slowly - as the frameworks do exist but are a lot harder to get the hang of. Once this is done - mass parallelism is in everyones hands. You are correct that a CUDA or STEAM core isn't as generally powerful as a RISC core - but its access to shared, high speed memory and ability to do specific transforms really fast, in a massively parallel way is stunning once it kicks in. If our astronomy programs were translated to shift load across a modern CPU / GPU platform - you would simply have sub-second responses for all operations - one day maybe.

One day a bright Uni student may come across a simple way to re-program on the fly from procedural languages to GPU languages - then over night your programs could be re-compiled to run on this hardware (not optimally maybe - that's harder) but it would run a lot faster. I saw this in the late 90s when one student wrote a translater from Fortran to C and recompiled Cayley (a 10,000 page Fortran program designed to do "infinite" precision maths on groups of over 10 ^ 60 elements - think ~ number of atoms in the visible universe - and got it all working over one weekend). Imagine if someone did that for general compute algorithms - game changer... (http://research.ijcaonline.org/volume82/number3/pxc3892109.pdf or http://au.mathworks.com/company/newsletters/articles/gpu-programming-in-matlab.html?requestedDomain=www.mat hworks.com#) - one day!

[sil]

There are none, as a partially disabled person I am painfully aware of such organisations and their actual benefits, around here unfortunately little and none would pick up donations. I do give my excess gear to families who can actually use them. The mac has no monitor an needs an adapter to connect to normal displays which I havent sorted out yet. It is a shame, its a cute machine but yeah its oldish and I dont know how it runs. So until I am able to sort it out it stays out of my way, its not going to the dump unless its burnt out and it all looked good when i popped the cover.


(sorry for going off topic guys)

[Camelopardalis]

And I guess I'm pointing out the obvious here but modern versions of Linux and Windows should install and run natively on it ;)

[sil]

Yeah but already have a ton of linux and windows boxes, never used Mac and have no need or interest for yet another computer. Would like to have my Amiga and SGI machines set up but got no room as it is. If it was a macbook though I would but it isn't and theres nothing tempting my interest.

[Camelopardalis]

Well, like I said originally, there's _different_ software on the Mac for astro, just not necessarily better or worse.. :lol: :lol:

Amiga and SGI...now there's a nerd after me own heart :lol: I used to have a bunch of both of those until I was forced to thin the heard when moving here :sadeyes: the reality is neither have the horsepower to run a web browser properly these days, and the fruit pies and NUCs and so on have more than enough geekery packed into a tiny box these days to make them great for astro.

[gregbradley]

I ended up with an i7 4ghz 64gb ram 480gb ssd where pgms are stored and 4 TB Hard disk and a Nviida 2gb video card.

Wins 10 now takes about 20-25 secs to boot up compared to 4 -5 minutes and 20 x 32mb images load easily and run like 4 would have before.

Photoshop takes about 10 secs to load instead of 2 minutes.

Its good as I have a backlog of image data to process that was taking too long before. Now its more confrontable. My newer camera a FLI Microline 16200 has nearly 32mb files and so does the 16803 so large files
seem to be a thing of the future and there is no getting away from that.

Greg.