* *How would you build a system specifically for video / image processing processing?

[g__day (Matthew)]

I asked this on a PC forum - but interested in what astronomers would plan it out. If I wanted to build a PC - high end - for image processing - what ideal set up would folk recommend?

At present I use a 2-3 year on quad core Conroe 2 chip on a motherboard with 8GB of RAM and a ATI HD5850 video card . I don't believe any of MaximDL, Deep Sky Stacker or Adobe CS4 is GPU enabled where it most matters for processing speed up - so I guess I'm processor bound.

In that case is the ideal set up a I970 (So 6 core = 12 threads) plus say 24 GB of RAM the optimal solution for faster image processing? At present DSS can take 2-3 minutes to stack 30 shots of 10MB (RAW) each - and some CS4 astronomy scripts take 3 minutes to process (e.g. Enhance DSO and diminish everything else).

Interested in folks thoughts.

BTW - For video encoding - ATI's GPU based video encoder - seems way faster and higher quality that a 2.3GHz quad core processor.

[mithrandir (Andrew)]

Money is no object? You can get several systems each with 16 quad core Xeons and 256G RAM. Rack them up and distribute the processing across them.

Andrew

[supernova1965 (Warren)]

I would get a MB with Crossfire or SLI it allows you to have two Graphic cards of the same type linked so they work together to process your images

[multiweb (Marc)]

I got an I7-950 with 24GB RAM, Win7 64bits and an NVIDIA card. That's plenty. If you need more grunt with realtime video display and interface refresh you can get more cards with a SLI bridge so you'll need a SLI certified motherboard. SLI will accelerate your realtime rendering display features. Ideal for first person shoot up games IMHO. But the grunt for processing power is your main CPU and RAM, not the SLI feature. Also keep in mind that a SLI setup needs proper ventilation and spacing, adequate power requirements too.

I have used PS CS5.5, After effects CS5.5. and MAX 2012, all 64bit apps running on the system. Those applications take advantage of the GPU.I haven't had the need for another video card really, even with realtime display using hair/fur modifiers or displacement maps. That usually will bring any graphic system to its knees so it's a good test. PS will speed up with display refresh/zoom , interface refresh,etc... After effects, well it's a no brainer. Any video app needs some breathing space. At the end of the day you'll still need a fairly fast HD access for swap drives and a lot of RAM. Other than that, it's pretty much it.

About $2k for a system like this including 3x1TB. Ball park.

[issdaol (Phil)]

Critical components are RAM, CPU, GPU and Hard Drives.

So get the best of each that you can afford on a good/solid Mainboard.

Reason to not forget hard drives is that the information still needs to be read/written and paged out to disk so when choosing hard disks for high performance go with several of the fastest, high speed high cache drives.

Going with very large capacity drives limits your r/w's to a fixed number of channels, heads and spindles for a given capacity whereas going for multiple smaller drives spreads the load and allows you to do multiple r/w operations that do not affect other r/w operations. You may be surprised by how changing hard disk setup can breathe new life into a system.

Also if you are trying to achieve realtime video edits/transitions/titling you may want to consider Canopus/Grass Valley specialist GPU products. These come with plugins for Adobe or you could use the excellent EDIUS app.

[g__day (Matthew)]

Andrew - I wish - if you want that approach - just go Zencoder and rent theirs actually!

Warren - SLI or Crossfire 2, 3 or 4 way works where the software allows GPGPU based processing. I don't believe MaximDL, DSS or the critical function of CS4 (at least) really use GPU capabilities well. Noel Calbronai (sp) astronomy actions grind my machines when I select the script Enhance DSO and diminish everything else - takes around 3 minutes to apply that action each time.

Marc - haven't played with SLI since late 90s and early millennium. CS5.5 I haven't researched but sounds interesting. Have you tried Noel Calbronai's Astronomy actions - particularily Enhance DSO? That would be a strong indicator of whether load was being shifted to the GPU/s. I would expect the first applications to offer real GPGPU acceleration will leverage Nvidia videocards first - simply because Nvidia has put more into CUDA to make it easy as a high level GPGPU programming environment than AMD/ATI has put into OpenCL / Direct Compute.

Phil - All good thoughts - the HDD for video edditing could always be SSDs or hybrid PCIx - SSDs if you really wanted to drop coin. I'll read up about EDIUS apps and add on - they sound really interesting too!

Meanwhile I'm urging the DSS guys to consider GPGPU enablement in a release future. Otherwise I guess its the CPU / number of cores that will most influence speed - all other factors considered!

Thanks all, Matt

[issdaol (Phil)]

I used to use Matrox and Adobe as my RTE suite but after testing out Canopus and EDIUS on the same projects/streams the raw performance, features and productivity of the setup convinced me to swap.

good quallity SSD's are great but be careful of lifetime compared to traditional magnetic HDD's. There is some new stuff on the horizon but probably too far out for your need and it will be very expensive.

[multiweb (Marc)]

Originally Posted by g__day

Marc - haven't played with SLI since late 90s and early millennium. CS5.5 I haven't researched but sounds interesting. Have you tried Noel Calbronai's Astronomy actions - particularily Enhance DSO? That would be a strong indicator of whether load was being shifted to the GPU/s. I would expect the first applications to offer real GPGPU acceleration will leverage Nvidia videocards first - simply because Nvidia has put more into CUDA to make it easy as a high level GPGPU programming environment than AMD/ATI has put into OpenCL / Direct Compute.

When using Carboni actions, most of them duplicate layers then upsample before running filters,etc.. only the final layer is resampled down so all you need is a lot of RAM in your system. I haven't had any problems yet. GPU/SLI/Crossfire wil only improve refresh rate in realtime applications, not so much for PS layering or processing stacks. Realtime rendering is also present in all high end video suites for realtime rendering and compositing multiple layers. If you are looking at this kind of machine (like AVID or others) then you need to go another way as there are optimised hardware platforms for this kind of apps.

[g__day (Matthew)]

So Marc - thanks - I'd class 24GB as a lot of RAM - when you select Enhance DSO - how long does it take you your main system for this command to complete - very interested in your answer?

Also if you try that again but reduce the amount of RAM CS5.5 can operate in - say by 4GB increments - how does this affect the time required for Enhance DSO to complete? That would be sterling to see how from 4GB - each increment of 4GB you add influences processing speed on that script.

Many thanks and cheers for any data you can provide,

Matt

[multiweb (Marc)]

Originally Posted by g__day

So Marc - thanks - I'd class 24GB as a lot of RAM - when you select Enhance DSO - how long does it take you your main system for this command to complete - very interested in your answer?

Will have to check and time it.

Originally Posted by g__day

Also if you try that again but reduce the amount of RAM CS5.5 can operate in - say by 4GB increments - how does this affect the time required for Enhance DSO to complete? That would be sterling to see how from 4GB - each increment of 4GB you add influences processing speed on that script.
Many thanks and cheers for any data you can provide,
Matt

You mean limit the amount of RAM that PS can access. I don't know if you can do this. You can decide which scratch disk it will use and how much of the total system RAM + Scratch disk available it can use but I never actually looked into reducing its usage. More maximising its resources access. Will look into it.

My guess is that if it runs out of RAM then it'll swap to drive and then you're back to needing a very fast I/O with your HD. RAM is like aperture. Aperture always win so get as much as you can.

In AfterEffects I render to RAM only. I max out 24GB for 6min of video sections (and that's plenty) but it's so fast. So very convenient production wise.

[g__day (Matthew)]

Marc,

Under CS4 - Photoshop -> Edit -> Preferences ->Performance -> Memory Usage (Just above scratch disks) One may specifiy the amount of available RAM and how much to let photoshop use. So in 32 bit Photoshop I can use up to 3.5GB RAM (defaults to 50%) and in 64 bit Photoshop I can use up to 7.2GB on my 8GB RAM system.

Matt

[multiweb (Marc)]

Originally Posted by g__day

Marc,

Under CS4 - Photoshop -> Edit -> Preferences ->Performance -> Memory Usage (Just above scratch disks) One may specifiy the amount of available RAM and how much to let photoshop use. So in 32 bit Photoshop I can use up to 3.5GB RAM (defaults to 50%) and in 64 bit Photoshop I can use up to 7.2GB on my 8GB RAM system.

Matt

Yeah but if you exceed the treshhold it would start swapping to scratch disks automatically so you'll kill your performance. You said in the test to limit the amount of RAM in increment of 4GB and see how it behaves. My guess is that it'll swap to drive (earlier) automatically depending on how much RAM the shot you've run the action on is using initially. So whether you have 8GB or 24GB of RAM if your carboni action chews up 6Gb let's say then the result will be the same because you're still within the RAM quota allocated.

[g__day (Matthew)]

Marc,

Agreed - its just on the tiff's I use from stacking DSLR RAW shots under DSS - I'm not sure how memory affects performance of the actions once you get above 8GB.

Matthew