Go Back   IceInSpace > Equipment > ATM and DIY Projects

Reply
 
Thread Tools Rate Thread
  #1  
Old 15-07-2015, 03:52 PM
griz11 (Dan)
Registered User

griz11 is offline
 
Join Date: Nov 2014
Location: Granite Shoals Tx. USA
Posts: 45
Desktop supercomputer for computational Astronomy

I supported some large scientific computing clusters when I worked for Digital some years back. Started reading about clusters built from small embedded system boards. Had an idea or two on how to use one for astronomy and a robotics project I want to build so I built one. The idea was to use the built in FFT program in the shaders on the GPU's to do calculations. They aren't well documented and not that fast so I kept looking for something to do the math. Ran across the nvidia Jetson TK1 board. Its a 4 core ARM 32 bit processor with an additional lo power core that is switched in to save power when its not in use. Nothing special the Pi's have basically the same thing. The difference is a K1 kepler GPU with 192 cuda cores on it grafted into the die. 600Gflops of math crunching ability. I want to use FFT's in a way similar to speckle interferometry to negate the seeing conditions and move my focus based on measurements of the star when seeing is subtracted. It will take more than one image 10-20 small frame 1024x1024 or 2048x2048 so it will be quick. Enough for focus since you only need a single star image. Worked out a routine from a tutorial on the mathematica site to take an image and split out all the stars into nice bite sized pieces for the computers to crunch on. So I'm going to use that if I can't get a single star image into the frame. Tested out a dummy 1024x2048 image on a single TK1 last night and it was 39ms to complete the calculations. Fast enough to do the burst. And with both of them active that will be reduced approximately 35-40%. Needless to say I was pleased when I saw the numbers. Another use for the TK-1's I have is to turn one of my mobility scooters into a self driver with voice commands. It will be awesome to have when I go to the races to shoot some pics. The system runs MPICH2 for the cluster software along with Hydra which is part of that package from Argonne Labs here in the states. Pretty much the same software they run at CERN and the other labs with huge clusters. The CFD computers the F1 teams use are just a bunch more K1's Several thousand to be exact. Don't know if the focus stuff will pan out but I have lots of other uses for the equipment so no sweat if it fails miserably Found the Pi's for 22bucks apiece. 12 for those Drok power supplies. It also functions as a power amp and volt meter.

Griz
Attached Thumbnails
Click for full-size image (GRZ_4539.jpg)
101.3 KB83 views
Click for full-size image (GRZ_4544.jpg)
92.7 KB72 views
Click for full-size image (GRZ_4537.jpg)
54.0 KB69 views
Reply With Quote
  #2  
Old 15-07-2015, 03:57 PM
multiweb's Avatar
multiweb (Marc)
ze frogginator

multiweb is offline
 
Join Date: Oct 2007
Location: Sydney
Posts: 22,080
Wow! Sounds like an exciting project. Make sure to post results.
Reply With Quote
  #3  
Old 15-07-2015, 05:30 PM
Atmos's Avatar
Atmos (Colin)
Ultimate Noob

Atmos is offline
 
Join Date: Aug 2011
Location: Melbourne
Posts: 7,013
That sounds incredibly interesting to say the least, really look forward to hearing how it all pans out
Reply With Quote
  #4  
Old 15-07-2015, 06:11 PM
Garbz (Chris)
Registered User

Garbz is offline
 
Join Date: May 2012
Location: Brisbane
Posts: 646
As a matter of interest you mentioned that you had trouble doing it on the GPU but then got excited about the 192 CUDA cores available on your little board. If you're using CUDA then what's the benefit of this system vs using the GPU?

What's the benefit over says a 2 year old Geforce 650 with 384 cores and 812GFLOPS? Or is low power a requirement?
Reply With Quote
  #5  
Old 15-07-2015, 07:22 PM
griz11 (Dan)
Registered User

griz11 is offline
 
Join Date: Nov 2014
Location: Granite Shoals Tx. USA
Posts: 45
Different boards

The Raspberry Pi has a videocore 4 GPU on it. That is what I was trying to use. Its not an open hardware piece so docs are sparse. I found an article on how to use it for FFT's in the shaders which is what I want to do but trying to implement it proved to be problematic and its nowhere near as fast as the K1. Has 4 cores.

So I'm going to use the PI's for other things. One possibility is a Hadoop cluster which is used for big data sets. Like a bunch of images. One of the boards I took out of the original stack is now running my backups. Its running freeNAS and is connected to a small raid array. The only truly open gpu is on the beagleboard black. But again no where near the power of the nvidia chips. Nvidia has the best software support and tools as well. All kind of libraries for the Tegra. OpenCV which is the standard for computer vision applications. Cufft lib for doing fft's. Several other math orientated libraries. OpenMPI for clustering with cuda support.CuDNN deep neural network libs for maching learning. Used in self driving bots. The little beast is doing some pretty awesome things. The automotive version has two tegras on the same board. Check out some of the stuff they are doing with it here. http://www.nvidia.com/object/drive-px.html The computer used to do this is basically two TK1's on a single card. Their site is full of amazing things people are doing with GPU hardware. Low power 12v operation compactness. Its going in a box sitting at the telescope. I'll communicate with it via X-windows over wifi. Its going to house all the programs for moving the scope shooting images etc too. And I'm going to use it on another robot project I have been thinking about doing for some time now. But to answer your question the older hardware is fine I have an older board I sometimes put in with the 660TI for more cores. There are some limitations as to how far you can go back.

On the PI side each one will have a specific task and the rest of the cores will be in the pool to be used as needed. I have the wireless touchscreen handpad working now good to 100m and the dew heaters and sensors for that. Just the focus left to do and I want to use a variation of Speckle interferometry to do that. Also you can use FFT's just like you do FWHM for determining focus. The two outputs will come closer as you get closer to focus and widen up as you go away.

I wouldn't say its for everyone the admin tasks alone are enough to scare most people off but if someone had a distribution they could download to one it would be much easier. Just getting all the programs downloaded and compiled has taken a couple of months. I can't put in a lot of hours a day on it and some days I just don't have the energy. So someone a little younger with more energy could probably whip it together in a week.

I posted up some pics of it on another forum and ran into the guys doing the Speckle. I didn't know about it just through getting on the Mathematica site and looking for ways to do FWHM I ran across using FFT's to determine sharpness.

I went with the PI's at first because they come with Mathematica on them. It has a ton of functions that directly relate to astronomy. You can move your scope and image with it if you want. I found a tutorial that has some killer functions to deal with images. One will split out every star in an image into a separate image for instance. Once they are in Mathematica they are already in an array you can operate on. Not being a math whiz I was leveraging my meager skills with the software. But to do a burst of images in a short enough time not to get in the way just required a lot more computing power than the PI's had to offer.

I bought a notebook less than a year ago and the dew has already messed up the keyboard. So I want something that I can put in a weatherproof box and not worry about it. I live right across the street from a lake its always dewy here.

So that is basically the story of how it all came about. Now I have some people to lean on for the math and I'm benchmarking the Jetson so they can decide if its right for their project in return. Just a couple more things to set-up and they will be able to log in and work on their stuff as well. They are 1024x1024 2D kernels so anything over 1024 cores will be optimum so you can run one element of the calculation on each core. Say it takes 10ns for a calculation and you are doing 1024 of them that is 10240ns for the total. With the GPU and enough cores it will take exactly 10ns for the whole thing. The test FFT I've been running is a 2048x2048 2D using 32 threads. Just using the sample programs I haven't even started to read up on the code. Fortunatly nvidia has a 6 week course on their site free that will teach you parallel programming with some interesting projects along the way in image processing. So I've been going through those to pick up the details. So far its a very nice course. I've only gotten through 2 lessons so far. The TK's came in Sat. Took all weekend to get the linux set up on my pc and flash both TK's and get them set-up. Linux is new to me I'm a BSD kinda guy. Linux was just getting popular when I quit working. But its starting to make sense now. Lots of new things new ways to configure networking displays etc.

The pics are the three screens on the handpad. It has a config screen as well. The sensors and handpad and the Mathematica program figuring out the centrioids in a star field. Its worth the cost of a PI just to get Mathematica in my opinion. Just make sure you get the PI-2 with the 4 core cpu. However there are free programs like scilab that are almost as complete you can run on any box.
Griz
Attached Thumbnails
Click for full-size image (Env-Dew.jpg)
129.3 KB46 views
Click for full-size image (Focus_Screen.jpg)
126.0 KB47 views
Click for full-size image (Mount_Control.jpg)
123.4 KB44 views
Click for full-size image (Working-Wireless.jpg)
70.2 KB55 views
Click for full-size image (MM-centroid-finder.jpg)
40.2 KB46 views

Last edited by griz11; 15-07-2015 at 07:58 PM.
Reply With Quote
  #6  
Old 23-12-2015, 01:57 PM
griz11 (Dan)
Registered User

griz11 is offline
 
Join Date: Nov 2014
Location: Granite Shoals Tx. USA
Posts: 45
I ended up doing something completely different with the Jetsons. I used two of them to power an Autonomous Rover. They have been switched out for one of the new TX1 64 bit jetsons. Enough power to run AI stuff not enough memory on the TK1 Jetsons and you can't share that on a cluster. The primary sensors are a Zed stereo depth camera and a spinning lidar I came up with. Also has an IMU GPS and a couple of other cameras light sensor and the standard environment stuff temp and humidity. One motor and encoder per wheel its a differential steering system. Its been a really fun build. I've had to really spend a lot of time brushing up on Linux and learning ROS the operating system for robots. This is the first time I've come up for air since I started on it. And I'd be working on it now if it weren't packed up for a move next week. Lost my extremely dark site but where I'm moving is a small subdivision out on the edge of town so I'm hopeful the pollution won't be too bad. There is a club that has a dark site too not as nice as just setting up in the yard.

All the parts are off the shelf stuff from Actobotics. The only fabrication needed is for the base and top acrylic plates. Just got up to speed on 3-d cad so I'm doing a 3-d model so I can have those laser cut. I just bought some thin stuff at the local store for prototyping it.
Griz
Attached Thumbnails
Click for full-size image (GRZ_1954.jpg)
136.6 KB25 views
Click for full-size image (GRZ_1947.jpg)
200.2 KB22 views
Click for full-size image (GRZ_2002.jpg)
95.9 KB23 views
Click for full-size image (GRZ_1996.jpg)
67.3 KB20 views
Click for full-size image (GRZ_2009.jpg)
160.9 KB18 views
Reply With Quote
  #7  
Old 23-12-2015, 04:54 PM
Atmos's Avatar
Atmos (Colin)
Ultimate Noob

Atmos is offline
 
Join Date: Aug 2011
Location: Melbourne
Posts: 7,013
Looks brilliant Dan, love the driver!
Reply With Quote
Reply

Bookmarks

Thread Tools
Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +10. The time is now 11:16 PM.

Powered by vBulletin Version 3.8.7 | Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Advertisement
Bintel
Advertisement
Astrophotography Prize
Advertisement