Pondering a Workstation build for astro imaging processing - peoples experiences!

[g__day]

This year I would like to build a rig to handle astro-photography image processing workloads (which means it could likely also handle games).!

Astro image processing can require stacking of 100s of shots per night - each shot could be 10 - 40MB - and there is a lot of mathematical processing (that no software that I know of off loads to the GPU). So I was pondering getting a 10 Core Intel beast and doing a rig built for Astro first and games second.... when a thought hit me that I first entertained a decade or two ago but never acted on... what about a dual CPU server board? Then a simple google and youtube search showed me several folk have gone down this path and got great results - at a fraction of the price I was expecting - simply because old Xeon chips sell for about $200 for a 8 core 2.4 GHz chip.

Now I only remember high end Tyan motherboards (like the S7070) being dual cpu - but a simple search shows many people are using the Asus Z10PE-d16 https://www.asus.com...ds/Z10PED16_WS/ (https://www.asus.com/Motherboards/Z10PED16_WS/) - which surprisingly supports DDR4 RAM (16 slots - up to a Terabyte of RAM) and has 6 PCI-E Gen3 x16 slots and can support NVidia in 3 way SLI or AMD in 4 way cross fire.

These boards sell in Australia for about $850 - or you might find dated rigs on Gumtree build on these boards (variant unknown) between $300 to $2000.

So a check on youtube and gumtree shows folks selling old Xeon CPUs that are 8 core, 16 thread, meaning this sort of build will give you a 2.4 - 2.6 GHz CPus with 16 cores and 32 threads to handle workloads that like many core solutions. I am yet to see if anyone astronomer has used the programs I like Deep Sky Stacker (freeware) or CCDStack on a multi core monster - but DSS does seem to keep all 4 core on my (or my kids latest i7 quad core machines) maxed out.

So my long winded question. What advice would folk have. I can guess that this rig could play games - but it would act much like a 4 core machine throttled at 2.4 - 2.8 Ghz even if it was running sat a pair of 1080s in SLI. I would have to check everything - OS (win10 x64 is supported) - memory types, Powersupplies, drivers - absolutely everything would work. Again there are folks building these rigs and they scream on 3d rendering that can use all the cores and they are decent at games (paired with a NVidia 980 or Titan card/s normally).

Have folk found - is your image processing CPU, memory or I/O bound?

What set up would be optimal for Image processing?

[pluto]

I do a bit of processing on my workstation at work (for 3D VFX/animation/comp) which is a dual ten-core Xeon (E5-2650v3) - so with HT that shows as 40 cores, 64GB ram, win 10.
I use PixInsight*1, DSS, RegiStax, Autostakkert!, Startools, Photoshop, etc.
Obviously it's very fast for software that uses multithreading efficiently :D

Personally if I was building an astro processing box, with my own money ;), I would just go with a current I7 over an older dual Xeon box. I think you'll probably get similar performance with a current I7 and a dual 8 core Xeon (just a guess - depends on which Xeon and age of course) for a similar price and in that case I'd always go with the more recent technology*2.

I should also note that I do the majority of my processing on my lappy (2013 XPS15, I7, 16GB) and I've never had anything that I could do on my workstation that I couldn't do on my lappy.


*1 when I first got this workstation I was still running Linux (CentOS) and PixInsight would only use all the cores from one Physical processor - so it would report 20 cores whereas on the same box on Windows it reports 40 cores... weird...

*2 we had an older Xeon that we had to retire recently because it didn't have SSE2 and the renderer we use had an update that required it - took me ages to work out why it wasn't working on just that box!

[sil]

I don't think ideal exists and the throttle is the person using the machine.
Putting any software on a Windows machine immediately slows its performance and uninstalling that software won't give you back performance.

Pick a purpose AND STICK TO IT. In your case you want a machine for gaming and a machine for astroimage processing. Therefore you have TWO purposes. How I've done this in the past for the CG industry is you have two OS hard drives, put your games and office crap, one a third drive and your CG apps on a fourth, a fifth drive is ultra high speed type with two partitions (understand the different between a drive and a partition then reread my post) which you dedicate to putting page files and temp/working space for each OS drive. So these days you use SSD drives for each of these and the point isto reduce I/O queues due to the software wanting to read part of a hard drive and the OS needing to read from a different part of the same drive plus the pagefile wanting to update another all at the same just because you pressed "Open file" in a program. So you gain performance plus stability by separating them. The page file drive you should fully format periodically to ensure it performs well and hasn't developed errors from stress. You don't need TB for an OS drive or a Programs drive, 240GB is plenty and you want performance. Add TB drives for temporary storage, but a RAIDED pair Could be used for where you put your source frames and generated registered etc frames too as you can eat up drive space fast, but again reformat periodically.

Anyway I yet to use my dual gfx cards for anything in my atrophotography so use whatever cards you want but not all software will use multiple cores/cpus so I would say go with fast cpu (higher GHz). Age of CPUs will matter, we've had 3.0GHz (not overclocked) Intel CPUs for 15yrs so a current generation 2.4GHZ i7 I bet will give you better performance than a 5yr old 2.4GHz Xeon plus will.

Money no object, I would get dual cpu of whatevers todays best intel cpu is, not necessarily the fastest but close with most cores. Simple way to calculate would be Speed * Cores = processing power and rank them that way choosing the best ranking power.

RAM and GFx cards are easier to upgrade reliably down the track, so for initial outlay you can save there.

[g__day]

The cores most people going the dual processor path use tend to be cores happy to run at 2.8GHz - so that is 16 cores and 32 threads (virtualised). Why this build is so attractive is these CPUs can often be sourced for $200 - 8 core Kaby Lake CPUs are faster clock speed but a lot more expensive! The specified dual CPU motherboards are expensive (brand new around $900) but they have 8 PCIE 16 slots and DDR4 support, plus excellent independent and fast memory and I/O lanes and M2 support. You could have up to a terabyte of RAM if that were warranted.

My current rig is an old Conroe Core2 Quad chip at 2.4 Ghz - on an old Gigabyte P35 board with 8GB RAM, 4 SSDs and 8 SATA-2 drives and a NVidia GTX 1070 graphics card. It runs a 30" UHD screen plus two 24" screens (in Portrait) configuration quite well. I don't play games often but it handles them fine.

What is slow is Deep Sky Stacker. Even running all files on SSDs the processing load max's out the CPU. Trying the same work load on my sons latest i7 with 16GB of RAM and a 512GB SSD isn't that much faster - say 30% - so I intuit the software is CPU bound.

A 16 core solution that is economically priced is worth investigating I imagine!

[rmuhlack]

have a look at these Pixinsight benchmarks. Will give you an idea of the performance of different hardware configurations for astro processing

http://pixinsight.com/benchmark/

[g__day]

Interesting to see a Xeon E5-2695 at 2.4 GHz leads the field by a long way, followed by another Xeon, followed by a I7 6700K at half the benchmark of the first placed Xeon!

[pluto]

It's worth noting with those PI benchmarks that a very significant portion of the final result seems due to the swap speed.
This seems a bit strange to me as I don't think I've ever used anywhere near my full 64GB of ram in PI and so it's never had to use swap - which is slow on my box and pulls my final score down.

Here are the numbers for my box (this was an old one in Win8.1, can't find my CentOS score and I've never bothered doing it again in Win10):
Total performance ...... 5157
CPU performance ........ 8683
Swap performance ....... 1929

So going purely by the total score you'd be better off using an i7 with seriously fast swap when in reality, as my box never seems to run out of ram in PI, my box will be faster.


I do think it's time for another SSD upgrade though! ;)

[g__day]

So Hugh - if you created a 32GB RAMdrive and ran your test again from there - what performance numbers would you expect it could deliver? Not even raid 0 960 Pro would be as fast as a large RAM Drive to hold all the I/O.

I would be very interested and thankful if you can try this at some point.

Cheers, Matthew

[pluto]

I'll give it a go later, got too much on at the moment.

I did just discover this (https://pixinsight.com/forum/index.php?topic=7644.0) though which instantly gave about a 40% increase in PI swap speed!

[g__day]

Many thanks - and that is a great find!

[sil]

keep in mind if you're using the software and hardware properly no machine is ever fast enough. speaking from experience in 3D modelling/rendering since the late '80s.

long tasks in PixInsight, like registration and integration I just set up and let it go while I head to bed or work. Once I have an integrated or drizzle integrated file to process its where my time and attention is needed so registration and integration I will do on sets of images I've taken and leave further processing to later instead of devoting all my time and all my computers time to just one image from capture to final image before moving on, I usually have a bunch of images "on the go" in various states of completeness in my workflow. Eventually final images result, sometimes I rush through a simple processing workflow just to see the potential in my capture. Its all a constant state of experimenting and optimizing each step in the workflow...and wishing my machine was faster (so I usually watch movies on another machine next to me just to keep me occupied).

[Slawomir]

Interesting thread, in particular that I just run the script and my computer got a total of 172 points :eyepop: in PI benchmark score. It looks like I really do need to buy a newer machine...

[RickS]

PI "swap" space is not used for VM paging. It's where PI stores image data, including the backup copies of images used to implement undo/redo and image history.

Cheers,
Rick.

[Slawomir]

A quick update.

I decided to get a modest PC (i5 6600, 32GB RAM) to replace my archaic one that was giving PixInsight benchmark scores around 200...


For the new machine...

Performance Indices
Total performance ...... 3224
CPU performance ........ 5568
Swap performance ....... 1178

Not revolutionary results, but these indicate that the new PC should be about 15x faster than my previous, now retired PC. Happy with that.

EDIT: After creating 4 swap storage directories in PI the performance has improved significantly:

Performance Indices
Total performance ...... 4422
CPU performance ........ 5616
Swap performance ....... 2356

[rustigsmed]

maybe wait for the new amd ryzen 8 core 16 thread which is out literally in a matter of weeks.

[Camelopardalis]

Yeah, building a PC might just get interesting again with a bit more competition in the market. Intel has had the game to itself for far too long, so kudos to AMD for pulling their socks up and (hopefully) getting something out the door that is competitive.

[lazjen]

This is what you need: http://www.anandtech.com/show/11121/intel-xeon-e7-8894-v4-cpu-24c-48t-9000-usd :)

[g__day]

An update to this old thread as I am planning on going down this path shortly. I found two decent stores Zcomputers and TheServerStore in the USA that focus on Workstations - the HP Z820 in particular which seems a very decent beast. I am tossing up two configurations:

HP Z820 Workstation USD $1,824 (https://www.theserverstore.com/product.asp?itemid=99)
Processors: 2x Xeon® E5-2695 v2 2.4GHz Twelve Core Processors, (24C/48T) - $760.00
Memory: 64GB (8x 8GB) DDR3 Memory - $160.00
Primary Hard Drive: 400GB SSD - $180.00
Second Hard Drive: No Hard Drive
Third Hard Drive: No Hard Drive
Fourth Hard Drive: No Hard Drive
Video Card: NVIDIA GeForce GTX 680 XLR8 - $160.00
Optical Drive: DVD-RW
Workstation PCIe Expansion: None Installed
Operating System: Windows 10 Professional 64-bit Installed - $45.00

or

HP Z820 Workstation (https://www.theserverstore.com/product.asp?itemid=99) USD $1,604
Processors: 2x Xeon® E5-2690 v2 3.0GHz Ten Core Processors, (20C/40T) - $700.00
Memory: 64GB (8x 8GB) DDR3 Memory - $160.00
Primary Hard Drive: 400GB SSD - $180.00
Second Hard Drive: No Hard Drive
Third Hard Drive: No Hard Drive
Fourth Hard Drive: No Hard Drive
Video Card: AMD Radeon HD 8350
Optical Drive: DVD-RW
Workstation PCIe Expansion: None Installed
Operating System: Windows 10 Professional 64-bit Installed - $45.00


The dual 10 core 3.0 GHZ Xeons would probably run poorly mutli threaded applications (like games) better because of its 25% higher clock speed across all threads. So its a trade off between 20 vs 24 physical cores.


The supplier also confirmed it can run a single NVidia GTX 1070 or 1080 (not SLI because only Workstation cards - think NVidia Quadro cards have that enabled).


One thing I wasn't aware of is dual cpu machines take about two minutes from power on to the normal BIOS boot sequence appearing on the screen. Apparently they go thru a prorietary hardware okay check and nothing much appears on the screen until that is finished!


So from the lasy post in February - those 12 core CPUs at 2.4GHz are available for about USD $760 the pair!

[g__day]

Just a PS - I see folk in Pixel Insight benchmarks with insanely high scores on these rigs - some are achieving half their 15K plus scores from their I/O throughput - up their with Intel PCIE 750 SSD throughput - sure enough they are using multiple RAMDRIVES to get 2.4GB/sec throughput. But hey RAM is cheap so a 64GB RAM disk of larger is I guess a totally valid way to speed things along!

http://pixinsight.com/benchmark/benchmark-report.php?sn=I452K9Q14PNK8292QB16T 3UWPO9548R0

[ChrisV]

Do you need the video card?

I wasn't aware that GPU acceleration was implemented in pixinsight yet - and it seemed that they have been talking about it for ages ? If I'm wrong someone let me know.

And might be worth getting an M2 gen 3 SSD. Uber fast and coming down in price. But need appropriate motherboard

[sil]

Don't depend on gpu acceleration, plus nvidia and amd graphics cards have incompatible architectures for gpu processing and most software will only use one type.

Plus there is always a better faster cpu coming "next week". Ignore all that stuff. Buy a quality motherboard which can handle far more ram *today* than you think you'll ever need. Spend money here on good motherboard and cpu, latest gen will perform better than last gen at same speed/spec each gen has better optimisations. Not all i7s are equal or better than i3 or i5 but Intel is always better than AMD CPU. AMD make great cpus but not as accurate and in the gfx industry have a long history of letting people down because of lack of precision and reliability. Buy a good chunk of economical ram today and next year upgrade to faster better ram, maybe max what the board supports. ram is always coming down in price and going up in performance and the economical ram today will perform better than whatever is in your current machine. similiarly with gfx cards get one mid range card today and it'll be great for gaming and if astro software gets updated to support gpu then 50/50 chance you have a suitable card, if not and you swap go for dual high end cards.

Ram and gfx cards are easy to upgrade anytime and any low end new card today will probably outperform anything you are already running, so if you need to save money this is the way to do it, then in a year or so you can upgrade to whatever then is the highend . Run OS on raided SSD, install apps onto another raided pair of ssd and chuck in quality (not economical or green) HDD for big space, maybe another SSD for moving pagefile to and configuring software to use it as its "scratch space" so you can format it any time as it has no stored data and makes all your software using it just scream. I'd still recommend amd video cards over nvidia purely because from long experience, nvidia drivers are not too stable and i've never once had radeon drivers crash my machines, office/work machines I look after dozens of them.

[g__day]

The GPU is more for if / when I would like to play games. Mind you I am still analysing the USA seller's warranty terms and costs, whilst the Fedex imprt cost is around USD $420 - so pricey if it arrives DOA and I have to pay to send it back!

Friends in the know say compare it to the 16 core 4GHz threadripper builds that will be guaranteed to run handle multi core workloads and run any games quickly.

With a workstation there is real buyer do your homework - for instance a dual GTX 1070 or 1080 in SLI won't work simply becuase HP won't support it. I have time so off to do research.

I totally agree buy quality - my current rig has served me very well for almost a decade - based upon excellent high end gear choices and upgrading 3d Graphics card every 3 years and adding SSDs.

[lazjen]

I'm waiting for supply to settle a bit, but will be going for a AMD Threadripper setup - the 16 core 1950, 64Gb RAM, m2 SSD. I may eventually expand it to beyond 64Gb and m2 raid setup.

I think I'll reuse some of my current components like graphics card as I don't need it to be cutting edge at this time.

It's great we've now got an AMD option that's a far better choice than Intel for the equivalent price.

[rustigsmed]

you mention playing games - even considering sli, yes its a great work station but I'm not sure it will be good at playing at all. the threadripper for example struggles on everything bar ashes of the singularity. i disagree with sil on amd - yes it has been true for some time but ryzen and threadripper are game changers for productivity purposes - they are well and truly competitive again (but everything else i agree with).

but how much cpu power do you really need for image processing? an old i7 is quick enough, its not like you are 4k video editing...

if it were me getting a new pc with a balance of gaming and work station i'd go with the 8700k, ssds and fast ddr4 memory, the gpu would depend on your monitor resolution and its refresh rate.

cheers

[g__day]

Must say doing more research the Core I7 7820X or Core I9 7900X seem very powerful for a respective 8 and ten core CPU. I am not fully across AMD Threadripper performance vs value yet. A big rig modern Inetl PC with high core CPU and fast I/O (PCIE NVME) and say 64 GB RAM will cost in the high $4K to mid $5K depending on how you spec your machine. So it's worth planning well.

Similarily landing a dual high core Xeon Workstation and add NVME storage and big GPU will cost around $5,000. So part of me says going local and modern might make more sense!

[Camelopardalis]

It depends. Ryzen and Threadripper are more scalable and with more PCIe lanes have higher potential throughout. Certainly great to have some competition back in the market, as Intel chips have stagnated for years.

Image processing doesn’t really benefit from having the larger L3 cache that differentiates the Xeons from the i7, so it’s really only a core count thing. If you want many cpu cores, the Threadripper is a better way forward and for less money. If you only need 8 or 10 cpus, then you don’t need Threadripper. Needless to say, the new high core counts from Intel are almost a direct response to AMD...

For that kind of money, I’d be building my own to ensure quality and performance.

[g__day]

Quite a quandry - I love the idea of Threadripper 1950 - far cheaper than say a 14 core I9. Think I will settle with a I7 8720 eight core CPU as an all rounder then upgrade if I need to a monster I9 multi core beast if its warranted late next year. Wish there were more benchmarks of astronomers doing image processing with ZWO mono and color cameras on purpose built big rigs!

For a big rig - reckon the AMD path would save over $1,000...

[Camelopardalis]

PI has a benchmark function...

In my (limited) experience, the amount of time spent in PI seeing what works well on my data trying to make it look nice far exceeds the amount of compute time expended to get it there.

[g__day]

Just had a friend who is into high end PCs try registering and stack sixteen pictures in DeepSkyStacker. Seem registration is single threaded and stacking is multi threaded. I thried stacking simply a dozen light frames of NGC 5137 - a lot of stars so a torture test - the registering alone took an hour on my 2.4GHz quad core PC!

[Camelopardalis]

Registration is multithreaded in PI, took less than 10 minutes to register 144 subs from my ASI1600(mono) on my 5 year old i7.

[g__day]

Interesting! For comparison how many stars were in the subs you were registering?

[Camelopardalis]

I think PI picked up about 2000 stars, but it’s automatic...

In DSS, you want to limit your stars to ~200 to keep the registration times reasonable.

[RobF]

Do I recall the Pixinsight guys talking about using the GPU to boost performance at some point?

[RickS]

There was some initial enthusiasm but last I heard it wasn't a priority. Don't hold your breath ;)

[peter_4059]

Just completed an upgrade of our desktop pc last weekend with PI in mind. I've gone for an i7 7700k, 32gig DDR4 2400 RAM and multiple SSDs. The MB supports 64Gig of RAM and that might be a future upgrade. I didn't bother with RAID but that is also possible.

I'm still trying to pry my teenager off it to try the PI benchmark but hoping for a good result.

[RickS]

Crow bar? :lol:

[peter_4059]

Ski trip to Japan has higher probability of success.

[RobF]

Sounds like a serious impediment to processing performance Peter :lol:

[peter_4059]

Managed to get PI installed today and ran some benchmarks. This new desktop is quite a bit faster than my laptop :)

[g__day]

Interesting to see the score posted - I presume you are using 4 RAMDRIVES to achieve that higher score from the disk sub system?

Nice to see where PI places all the CPUs I am interested in - even if it doesn't reveal the clockspeed of the CPUs tested1

Out of interest how expensive is PI? Last I looked it was several hundred dollars to install beyond a 30 day trial licence - is that still the case?

[peter_4059]

Hi Matthew. I'm using two SSDs for the swapfile folders- four folders on each and the ASUS RamCache III utility. I think this works a bit like a RAM drive however you don't allocate a drive letter. It works it out on the fly.. No RAM drives. On testing this gave a massive improvement in the swap score with a slight CPU penalty.

Last time I looked PI was a few hundred Euros. Still a lot less than PS when you include all the addins you also need with PS.

[Camelopardalis]

PI is 230 euros, but worth every peny/cent. It’s so powerful. Something to grow into. Mathematical precision vs artistic licence really.

As another frame of reference, my dual core NUC with i5-7260u, 8GB ram and Sammy EVO 960 gum stick puts out CPU 3496 Swap 4108 Total 3601 Transfer 741.74MB/s

[peter_4059]

Matthew,

You can check out the build details on the PI benchmark site - just expand the category and click on the serial number:
http://pixinsight.com/benchmark/

Something that is pretty obvious is number of threads has a huge impact on CPU score. Swap score is largely influenced by presence of RAM drive or in my case RAM Cache. The highest swap score across any CPU is around 20,000 and this seems to be achievable regardless of the CPU chosen.

I went through the benchmarks and picked out the highest CPU score and plotted that against threads. The dip at 20 threads is an Intel i9 whereas the points either side are AMD Ryzen processors.

[lazjen]

Nice to see that for equivalent specs for the Threadripper, Linux canes Windows for performance.

[g__day]

Many thanks guys. BTW - I went I7 7820 with 32 GB 3000MHz RAM, and fast drives - tried registering and stacking 17 shots of Omega cluster in DSS - the new PC did it all in 2 mins 10 secs - the old one took just over a hour!

[Camelopardalis]

*cough* benchmark *cough*

:lol: