Your advice: How to store 20 TB of data (Raid Array) at home?
Dear fellow LESbians,
my collection of digital media files grew exponentially during the Covid-19 pandemic. I have a rented server with 4x10 GB spinning disks in RAID-10 (= 20 usable TB in total) and the array is already filled with over 10 TB. I expect to hit the full 20 TB in some months time, but this will then also be the maximum storage, that I will need.
However, the server is not horribly expensive, but also not a bargain with monthly payments around EUR 90,00. I wonder whether it would make sense to have the data at home instead. I have a home internet connection with 100 Mbit upload capacity, so even hosting my data at home for access on the go could be an option.
What would be your choice for the job? A NAS? Something else? I have no idea what the current "state of the art" solution is. It is important to me to at least have the data in a RAID-1 array. No Raid or Raid-0 is not an option. I know that a Raid Array is no replacement for backups, but it would let me sleep better anyhow. A NAS would be my first idea, but I saw that they are not cheap and I wonder whether going on renting the server (with free hardware replacements) isn't the better choice in the end...
Thank you very much in advance for your ideas and suggestions!
Stay negative or at least keep it mild & kind regards
Amitz
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Comments
Just regarding streaming from home:
I fiddled forever with Plex since Vodafone doesn't give you a public IP. I am now using our FritzNAS (Fritzbox + any external hard drive) and while it doesn't have the looks from Plex, watching movies/series from remote works like a charm. Just your remote device probably will need IPv6 (either from your mobile carrier or the network/wifi you are in when watching). Only really tried watching on mobile devices, but you can also access the NAS in the browser and watch
We only got 25 Mbits, so you should be good.
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
a bucket will work!
but seriously, how tolerant are you to heat/noise at home?
Webhosting24 aff best VPS; ServerFactory aff best VDS; Cloudie best ASN; Huel aff best brotein.
did you know that Amitz is warewolf ?
There are quite a few options, the best would be a NAS/Microserver as they tend to be quiet and have fairly low power consumption.
You can also reduce the noise and power consumption to an minimum by using the least amount of disks possible - here is a few options, all of which use the same base hardware a QNAP TS-451+.
The NAS alone costs £320 - this could easily be substituted with an old desktop PC assuming it has 4 SATA 3 ports and a gigabit NIC.
Disk wise if you wanted no redundancy you could go for a single Seagate Exos X18 18TB disk which is the quietest and least power hungry option to get almost 20TB in a 4 bay chassis at £341 for a total of £661
Add an additional Seagate Exos x18 18TB disk and use RAID 1 for redundancy for a total cost of: £902
You could also go for 4x Seagate IronWolf 10TB in a RAID 10 array for £254 per disk or £1336 including the NAS.
These are all expensive ways to get ~20TB with little redundancy but for a home/office environment it will likely be the best experience and use the least amount of power.
If power/noise isn't something that concerns you I would recommend getting desktop PC / Tower Server with at least 6 drive bays and purchasing 6x 4TB WD RED HDDs - this would cost £536 for the drives + the cost of the PC. You would then need to RAID them using RAID 5, not the fastest or most redundant but allows for a single drive failure and gets you 20TB. ZFS would be ideal for this likely using RAIDZ1.
Its almost always cheaper to rent as many providers purchase disks and servers in bulk and get a much better deal, it also means it is the providers responsibility to replace failed hardware which can prove expensive in the long run.
The ideal is to actually do both, keep a copy of your data locally and have an offsite backup.
I know what you are getting at, but I think that having a real server / not quiet computer at home would be outside my tolerance level. I really like it quiet...
I love you, but I wonder whether all this can also be done with a dog as I (by coincidence) have one at hand and am allergic to cat hair?
Hush! Hush!
Thank you very much for the summary concerning my options! I will do some calculation to see which of the possibilities would make most sense in my case and your advice is a great starting point!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
I did what @natvps_uk suggested just last October, going for the pc option as a proper NAS can be slightly limited in hardware and I needed it for some other tasks at home anyways, running a few VMs.
Went for a Fractal Design Node 804 case which can hold up to 10-12 drives.
I had 4x 3TB drives in my old NAS so just moved them over.
It cost me around 500 euros without the drives but leaves me plenty of room for upgrades when it will become necessary
@daffy: Excellent, thank you!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
I'd start by browsing the /r/datahoarder subreddit a bit...they're constantly doing this sort of half enterprise half DIY mass data storage stuff so will be on top of the current gear recommendations
I'd also check whether you can transcode some of your linux ISOs to x265. Should make for much smaller file sizes.
If there are part of the 20TB you don't often need you could look into cold storage options. Even if you offload say 5TB it would mean less disks
I'd probably go for a TrueNAS zfs setup...handful of drives and a SDD cache.
Btw, my biggest regret is that I was a bit el cheapo with the motherboard which only has 2 ram slots and filled them with 2x8GB.
After a while now I see more and more use cases for my little machine and would have done well with 32GB.
So consider your options/budget/usecase and don't make the same mistake I did
Great info, thank you @havoc & @daffy!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
I use 4x 12TB (connected via USB3.1 in dual bays) with ZFS raidz1 and 4x2TB ssd caching it all (L2ARC).
Works really fine to back a CDN and generate thumbnails. Gives me 36TB of usable storage.
@foxone - Great, thank you! I was wondering whether simply attaching 4 drives via USB to my existing Linux PC at home would be a viable solution, too! Is there really no issue with creating a RAID 1 array with external USB drives?
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Having just done this I'd tread carefully - take a careful look at the USB power situation. Multiple nvme drives can overload the USB's aggregate ability to feed it. Worse it only shows up under load since nvme draw depends on usage. Chances are very good that it "works" but is outside of spec
TrueNAS freaked out about data issues as a result in my case
I went with an Intel build years ago, since they support C6/C7 power states. When they're in a C6/C7 power state they barely draw any power, even less than most NAS/microservers. Threw in a a ton of second hand (consumer grade) hardware (CPU, motherboard, PSU, DDR3 memory and a small SSD for booting), and shucked a couple of WD MyBooks for helium filled WD Red-like drives (some even had plain WD Reds inside). It runs Debian with a ton of services in Docker (Plex, Sonarr, Radarr, Vaultwarden, Wireguard, etc.).
I have over 27 TB of (mostly) fully reduntant storage for a little under 1000 € (a 10 TB WD MyBook gets on sale on amazon.de for 159 € every so often). One of the drives has over 7 years of power on time, and still counting. Would've been much more expensive to rent this elsewhere (although that is by no means a fair comparison, since I'm running consumer grade hardware)
What I used to do is run a HP Microserver G7 for my NAS, but now what I do is just keep the drives on a shelf and drop them into a enclosure when I need to get at the data and index it using https://vvvapp.sourceforge.net/ (Virtual Volumes View). Works quite well if you don't always need instant access to the data and are just looking to archive stuff.
Cheap dedis are my drug, and I'm too far gone to turn back.
The more I think about it... Maybe I could do it really simpler (and cheaper). If I indeed convert all my Linux ISOs to H.265, then 10TB of usable disk space would be plenty enough. If I buy two external USB drives (something like 12 TB to be on the safe side), then I could fill the first one and mirror it via rsync every night to the second one. No raid, but still a comparable result. This solution would allow to just use the computer that is under my desk anyway and IF I would really like, then installing a media server on it would not be a big thing.... Mmmmh....
But this opens up a new question and I am hoping - again - for your appreciated feedback:
I see that most external USB drives of that size (approx. 10-12 TB) come from three manufacturers, at least on Amazon.
Those are Western Digital, Seagate and Intenso.
I wonder: Which of these brands is seen as the most reliable nowadays? Or does it not matter, as they are all equally shitty in 2022? Does anyone have experience with those "WD Mybook" or "WD Elements" products? It seems as if they offer more than I need. Two basic large USB HDDs with cases would suffice.
Again, thank you all in advance!
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Please do keep in mind that support for x265 playback is still limited, although it does save a lot of space.
WD Mybook and WD Elements both contain the same drives, they're whitelabeled WD Reds, and built for running 24/7. The only difference is the Mybooks come with hardware level encryption. Most drives are not built for running 24/7, especially not external drives, so I'd recommend going with WD's. I've shucked quite a few of them and they never failed on me. If you want to use the computer under your desk you might consider shucking them, as computer cases generally provide better cooling and less risk of tripping over USB cables, resulting in potential data loss.
I can agree. WD's have been really stable for me. All drives I used for NAS/Plex have been WD Elements so far. All still going strong.
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
I don't have made any experiences with WD's external drives yet, but with Seagates HDDs, I'm running currently 2x 4TB Expansion(+) Drives 24/7 in my home network and they are pretty good.
Greetings from 🇩🇪 North Rhine-Westphalia, Xenic.
this is probably the easiest approach and mirroring via some automated copy should be easy enough. also saves you the hassle with managing the raid and possible unneeded rebuilts being triggered by power loss or the like (which would try to recopy the whole data).
for the brands I'd say it does not matter so much anymore nowadays, which brand you choose, but rather check what kind of drive is inside. all brands have different drives for different use cases, like WD red green blue black and so on and these might even have different speeds (5400/5900/7200 upm), different cache size and eventually different technologies ( google: cmr / smr ).
from what you listed I think Intenso is not a vendor for disks as such but more likely a reseller who only puts its own label on it. so inside you may find different drives/brands sometimes even for the very same product. usually you can find some discussions or reviews about that from people who tend to buy external drives to dismantle and use the drive seperately (because those are often cheaper compared to buying internal drives).
You guys are wonderful and incredibly helpful, thank you very much! I will check the available free bays in my desktop computer to see whether I can fit two additional drives in there, otherwise I will go the WD external route...
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
no happy ending without the Amitz dance.
There you go:
Amitz, a very stable genius (it's true!) and Grand Rectumfier of the official LESLOS® (LES League of Shitposters).
Certified braindead since 1974 and still perfectly happy.
Why so complicated? Buy a used server on eBay where you can simply put in your drives and send it in for colo? Should be around 40-50€/m.
+1 for the MyBooks. Super solid and reliable, though you can mostly achieve the same result with a WD Red drive and a USB Enclosure if you want to save a bit of money. Can't go wrong either way.
Cheap dedis are my drug, and I'm too far gone to turn back.
This is a very common use-case over at ServerBuilds, and a number of whitebox NAS builds would be suitable. E.g., $40 LGA1155 board in a large case like RSV-L4500 (15 bays) using SAS2 HBA and either cheap SAS spinners ($10/TB) or WD shucks (MyBook/Elements), running Unraid (or snapraid+mergerfs if you like). The WD shucks are HGST datacentre drives and very solid. Unraid makes it very easy to incrementally add drives of mixed capacity.
Plex is offloaded to a $100 SFF desktop with a 9th-gen Celeron, using QSV hardware transcoding of 1080p source for remote users. These little corporate desktops can do 21 simultaneous 1080p transcodes while drawing only 10W. 4k content (remux) is in a separate Plex instance on the NAS with transcoding disabled (direct-stream takes almost no resources on the NAS).
Many of our users have over 100TB at home using these builds. Pricing above is for the US, of course; a number of our EU users opt for newer NAS builds with QSV. Feel free to join us on Discord; I'm a mod there.
RAID/Unraid is not backup; maintain automated 3-2-1, etc., as you well know.
I don't know if you are from the UK or not, but I'm with Vodafone for VDSL and have the 80mbps pro plan for £23 a month, they will give you a free static IP if you request one (I have one).
Thanks for bringing that up! Unfortunately, I am from Germany
When I last called them and asked, they said that they won't do this anymore.
Ympker's VPN LTD Comparison, Uptime.is, Ympker's GitHub.
I personally never use RAID at home. I like to keep things dead simple.
But then I barely have 5TB of data to keep. Not much of a sentimental person, I am.
♻ Amitz day is October 21.
♻ Join Nigh sect by adopting my avatar. Let us spread the joys of the end.