Data Storage and backup solutions... advice welcome

[joshman]

Well, now that i'm fully invested into pixinsight, i'm pretty quickly running up against the storage wall. in less than 6 months I've managed to nearly fill a 2TB SSD with my raw files and Pixinsight project files.

I'm looking at solutions to fix what is about to become an immediate problem.

I'm looking at getting a NAS system setup at home for local storage, as well as some cloud storage for offsite backup.

So I'm looking for advice on brands/vendors, both for NAS setups, reliable drives, and cloud storage solutions.

Actually, any advice that can be offered is welcome!

[ronson]

Hi Josh,

A quick one from me, just my 2c.

1. NAS - popular and reliable brands are Synology and Qnap.
2. Hard drives - popular and reliable brands are Western Digital and Seagate.
3. Cloud storage solutions - popular and reliable services are AWS, Azure, GCP and Backblaze.

Each one offers multiple options for various budgets.

Depending on your budget, you could also consider the 3-2-1 backup strategy: https://www.synology.com/en-global/dsm/solution/data_backup

[joshman]

Thanks ronson, I've been looking into Synology for NAS, Looks like i might be up for a bit of coin to setup properly, but, you know what they say; buy once, cry once. I'm hoping to go for a NAS setup that can give me some speed and data redundancy, I'm swimming in murky waters here, so theres a bit or learning to do.

Still a toss up between WD or Seagate for me, I've always bought Seagate, but I believe that WD has a lower failure rate.

Cloud storage is proabably the hardest one to choose, I want something that has good reliability and expandable storage, fast access, and well priced. even better if the server farm is onshore.

I think my intentions with the NAS and the cloud backup should get me a 3-2-1 backup without too much trouble.

[AstroViking]

I have a 4-drive Qnap array from about 2010 or so. It's slow, no longer supported, but continues to work well enough for what I want. With 4 * 3TB drives in a RAID5 arrangement I have about 8.5TB of available space and I've never managed to fill it.

If/when you do come to buying the disks, be careful of the magnetic recording method. (Assuming you go with spinning platters.)

The SMR (Shingled Magnetic R-something) is glacially slow due to the way the drive writes the data. Avoid them at all costs and only by the CMR drives.

The WD 'Red' NAS-rated drives under 3TB are SMR and CMR in the larger capacities. (I found out the hard way when one of my 3TB 'Red' drives died and had to to some serious research to find out which new drive was OK.)

If you're going with SSD, then this is totally irrelevant to you. (Just be aware that SSD drives have a finite number of write cycles before they kick the bucket.)

Cheers,
V

[joshman]

Thanks for the heads up Steve. I'm still undecided on the medium I'm going to use. I'm thinking I'll probably need to go with Spinning platter, unless the price of SSD's comes down dramatically, it'll end up being a price based decision. I don't need a huge amount of speed for the NAS, apart from media, I intend to bring projects locally to my processing machine to work on them., before spitting them back to the NAS once i'm done.
I'm looking at dumping a fair amount of storage into this NAS, somewhere in the order of 20+TB. I've nearly filled 2TB in the last 6 months, not to mention the several TB I have lying around from the years gone by.

[rustigsmed]

hi josh,


i ended up building my own home server using Truenas Scale operating system (proxmox is another option). Truenas Scale allows containers, setting up SMB / storage, virtual machines etc. I have 2x16tb hdd's in a mirrored configuration (plus a couple of ssds for non storage purposes). I store astro files on there, as well as media which I can stream anywhere in the house. I also run pihole (recursive dns server / network wide ad blocking), as well as nextcloud mostly for the purpose of backing up phone photos (no google/apple needed for cloud storage - just make your own cloud) (and have also run a Valheim game server from the system).


As I'm mostly on linux i have put aside one Windows 10 VM which is run through the server - with a gpu passthrough for any windows apps I might decide to use. Access to the VM is easy done through Sunshine (host) / Moonlight (client) applications (allows high refresh rate desktop access - 4k 120hz if your network can handle it).


The SMB shares mean any device on the home network can save to it or copy files to and from it. Anyway it was worth the effort, learnt a lot and I can add more storage / alter the raid configuration when/ifI need to. Just make sure you go for quality NAS HDDs and not the usual domestic HDDs for longevity - they write a lot. Also a cpu with a relatively high core count if you're going to be using it for multiple uses (I used a 5950x w/128gb ddr4 ecc memory, rtx 3090). I'll often find myself processing planetary on the win VM at the same time as my desktop pc to save time.


I'm not a comp dev or anything just a bit of an enthusiast and managed to get it all done without any assistance. Anyway - if you've got some time and are happy to tinker it could be an option.



Cheers


Russ

[jakob]

I installed a Terramaster NAS 2x8 Tb about a year ago and this has worked flawlessly.
Jakob.

[joshman]

Thats a solution and a half! Very inspired. But I don't have the willower, expertise nor time to dive that deep at the moment. I've been looking for a relatively simple, but tried and true storage solution to just centralise my home storage needs. I don't need VM or secondary processing etc.

I've settled on getting a Synology 4 Bay NAS, and at this stage I'm putting 3x 10TB Western Digital Red Pro drives into it. I'll set them up in a RAID5 configuration, so i have some level of fault protection. I'm not sure what i'll do with the spare 4th bay yet. I'm tossing up either a 20TB drive to mirror the RAID5 completely, or drop an SSD into it for serving up media files. If I go down the 20TB mirror drive route, I'll probably make it a hot swap so I can take it offsite.

20TB should keep me humming along for a little while. And I've already got a good amount of data to push across to it.

[Addos]

hi mate, great question i posed to myself recently. ive just gotten into the nas world with an amazon cheapy the synology ds220j (two bay). what i can say having gone through the experience is:

* bit of a learning curve with nas'. learning raid types, having to learn more about home networking, and the synology os.

* raid is great but having to spend twice the money for the base amount of storage is quite the price to pay for an extra redundancy layer. 2 8tb drives ontop of the price of the nas box and the exercise to protect my astro data cost nearly a grand.

* on the upside, very easy backup process once setup. going through the pain to fix bottlenecks in the local network and always on nas boxes mean i dont have to batch files from the mini pc on the rig to a usb stick to transfer to my processing pc. setup robocopy on nina to the nas network folder and when i wake up in the morning the files are already on the home network and can pull them across to the processing pc.

on the counterside if i'd gone for the external hdd option wouldve been far, far simpler, cheaper but with slightly less redundancy and utility. dealers choice!

[joshman]

I can definitely appreciate the Initial outlay in money. The solution i went with was not a cheap exercise. I went with a RAID5 solution, which stripes the data across 3 drives, with a layer of redundancy. Meaning I can tolerate 1 drive failing and still retain all my data. I also went with NAS rated spinning platters, so I'm hoping my Data will be safe for a good long while. I do plan on getting an offsite backup happening, either through some paid cloud service, or through manually backing up to a large external drive and moving it offsite.


I've yet to setup robocopy in Nina, but i plan to. i think i'll end up putting it onto an External SSD and manually copying it across to my HDD so i can sort and cull before spitting it across to the NAS. This'll also allow me to travel with the setup without needing to alter any scripting or sequences.

[Leo.G]

While I don't use it myself my son has a HP micro server (G8) which is very handy. I was tossing up between it and an NAS (a gift) for his own needs and we soon came to the decision the micro server was much more usable while still having the capacity of a small NAS.


https://www.auspcmarket.com.au/systems/servers/hpe-microserver-g10-v2-g6405-1p-16gb-1-2-sata-lff-3-5-nhpintel-vroc-raid-1yr-warranty/


While this isn't a cheap option these little units are amazing for the money if that's the route anyone should choose to take. They can be had cheaper shopping around new and also much cheaper used.


If you don't want the computing side a NAS is better than portable storage.



If looking at an used NAS (I buy specifically used due to finances (disability pensioner)) be wary any discs marked as good will likely be on their way out and figure in the cost of replacement hard drives. The NAS drives are generally a higher quality than standard computer hard drives:


https://blog.synology.com/xmas-wishlist-why-choose-nas-drives-over-desktop-drives-for-your-nas





I believe but I'll have to check with my son later (he runs his own servers and programs) that if you run any of the server versions on a dedicated computer external drives can be used in a RAID configuration with internal hard drives. For me that would be my running a dual boot system so I still had my wonderful Windows 7. My son runs several variants of server editions I believe (I know nothing about servers or programming because I don't want to know) over several machines.

[Windston]

+1 for truenas.

Pretty easy to set up if you have an interest in learning a little about server architecture, and very powerful.

I run not only my SMB file share, but a plex server for media anywhere in the house and through the internet when away from my LAN. Adblockers for getting rid of nasty trackers network wide, and VMs.

Being able to use an old PC that I had laying around meant that it only cost me the addition of a few HDDs.