where to store/backup all my massive raw video data on a budget

[black]

I was wondering where to store/backup all my massive raw video data on a budget? And I don’t want to use any cloud services.

[Oliver Daniel]

Lacie 2Big perhaps?

 

http://store.apple.com/uk/product/HB643ZM/A/lacie-6tb-2big-thunderbolt-series-raid-hard-drive-with-thunderbolt-cable?afid=p219%7CGOUK&cid=AOS-UK-KWG-PLA-PLA_UK_BTB

 

Sure, it costs a bit but you need a fat hard drive anyway ;)

[black]

hmm, thanks for the reply Olly but that's not really the on-a-budget approach I had in mind ;)

[Oliver Daniel]

Unfortunately, I think thats your only solution.  :unsure:

 

I have a 3Tb hard drive that cost me £120. Nothing fancy but stores lots. Grab a few of them perhaps? 

[dahlfors]

You won't find any cheaper solution than storing it on 3 or 4 TB hard disks, those are the cheapest per GB of data. Use those for backup / storage for different projects, connect and disconnect whenever you need to. If you got a tower computer, you might already have empty slots for disks, if not you will need to connect disks through usb/firewire/esata etc.

 

If you want more storage and more convenience in using it, use an old machine and add disks to that one and build a raid array - or do it the simpler way and get a NAS device, like a QNAP / Netgear NAS.

 

There aren't really any other options on a budget.

[ntblowz]

I guess buying lots of external HDD is the only way for RAW

[powderbanks]

I guess buying lots of external HDD is the only way for RAW

 

Well, if you're just looking for something to dump your footage to, yes, an external drive will be fine. But for archiving, backup or even working with the footage, it's less than ideal. I'd have an archive/backup copy of the files on a RAID 10 array and then copy files to a RAID 0 array for when I'm actually editing.

[Bruno]

I use bare hard drives and a dual hard drive dock to read them.
This way you only have one "reader" and one power supply, as opposed to having several external hard drives of different formats and with different power supplies.

http://www.everythingusb.com/startech-usb-to-sata-standalone-hdd-duplicator-dock-18699.html

You can also get these anti static hard drive cases that look a lot like a betacam tape case.

http://www.wiebetech.com/products/cases.php

Also, with the raw footage you might want to look into compressed raw formats for archival, or even some sort of prores depending how much you need to push the footage.

[black]

Hmm, I was wondering, if it is based on experience what some "experts" are stating, that a hard disk will go bad if it's not in use and just stored away in a dry place.

I mean, I re-used a 20 year old scsi hd from 1992 (from an old mac IIci) the other day and the thing was still working. The has-to-be-in-use theory I think, only applies to SSDs because as far as I understood it they always need a certain charge to hold the data, and that charge will fade away over time. (How long? I don't know.)

[Bruno]

What I heard is that you should plug them in every year for a day or so, not sure how accurate that is, but anyway hard drives keep getting bigger and better, and it's very likely that in a few years time you'll be able to copy a bunch of old hard drives to a single one.
I've had loads of backed up projects on Zip drives, CDs, DVDs, Blu-Rays that I recently copied to a single hard drive. It's very likely that in 10 years I'll be storing all those hard drives into either a new format or an extremely fast and cheap cloud solution.

[Sean Cunningham]

I just fired up a couple HUGE SYSTEMS HugeMediaVault DualMAX (SCSI-160) arrays a couple weeks ago, that likely hadn't been fired up since around 2004 or 2005 at the latest.  That was one of those "fingers crossed" kind of moments, plus hoping I remembered the proper cabling and terminator setup, plus hoping that the SCSI-ID dials hadn't been fiddled with in all these years and two cross-country trips.

 

Back-up is an enormous problem.  Hard drives are the only cost effective solution that are also fast enough.  For the individual or for facilities generating tons of data.  Tape technologies haven't kept up and they've always been a bad solution anyway, both hardware and software wise.  At one time "experts" expected optical discs to be viable.  LOL.

 

There is no truly longterm backup solution for digital data.  This is one of the biggest issues for longterm film archival and one that the MPAA might have truly screwed the pooch on when that d-bag Jack Valenti was in charge.  Nothing has been invented that has a chance of outlasting archival film, properly stored.

 

 

edit: I can't imagine the hell it would be trying to find a working exabyte drive, or metrum, or the crummy software that was used to back data up back then.  Getting ahold of an old machine with a SCSI or SCSI-II interface and the right version of an OS.  That stuff was awful.  ZIP drives didn't last a year because the drives themselves were crud and they'd just start corrupting any disk you put in.  Is Iomega still around?  Oh look, they're Lenovo now.  I know never to buy any of their stuff, ever.

[black]

maybe I should go with something like this (at a smaller scale)

http://blog.backblaze.com/2013/02/20/180tb-of-good-vibrations-storage-pod-3-0/

 

if I only could figure out what OS they are using ...

[Sean Cunningham]

My first and only guess would be linux.  Linux tends to be the best and most common OS choice for embedded, purpose-built device scenarios.

 

They just self-contained all of the points of failure you have with a dedicated PC + array.  By having the host PC inclusive to the arrangement they did remove the software+hardware related problem you get with a media only solution but you could very well be in a position where the media and data is good but the integral host PC has failed at any of the plethora of potential points that they do fail.

 

The saving grace with new drives and continually moving your data forward for fear of losing it (not only to failure but of planned obsolescence), is they get faster and smaller and cheaper, requiring fewer of the new to eventually landfill the old.  It's just a dreadful, repetitive and wasteful process.

[dahlfors]

Yes. The Backblaze storage pods run Linux.

 

Basically, this is what I first suggested for you: running disks in raid-mode in a normal computer case. Linux and FreeBSD works great for software raid setups in raid 0, 1, 5 and 6 modes. They are also among the cheapest solutions, since you don't need the newest hardware and it might work with old hardware that you already have.

 

However, it takes more knowledge to run a setup like this though. Buying an off-the-shelf NAS is for most people the more convenient option, mainly because it doesn't require as much technical knowledge to setup the NAS, and because it is easy to replace a failed disk - it's usually indicated with a light or in some simple way in the maintenance interface.

 

Personally I've used FreeBSD file servers for over a decade now. My current setup is an array with seven 2TB disks, which give me 10 TB effective space and I can lose two physical hard disks in the disk array before I lose data. Read and write speeds over the network usually range in the speeds of 75-120 MB/s, so it's quite close to a normal local disk in the machine. For smaller files it isn't as fast though.

[dahlfors]

Hmm, I was wondering, if it is based on experience what some "experts" are stating, that a hard disk will go bad if it's not in use and just stored away in a dry place.

I mean, I re-used a 20 year old scsi hd from 1992 (from an old mac IIci) the other day and the thing was still working. The has-to-be-in-use theory I think, only applies to SSDs because as far as I understood it they always need a certain charge to hold the data, and that charge will fade away over time. (How long? I don't know.)

 

The older a hard disk is, the more likely it is to fail, whether it is in use or not.

 

The zeroes and ones are stored on magnetic platters, and on a hard disk that isn't in use, the directions of the magnetization will weaken over time, eventually so much that your data can't be read properly.

 

And then there's the mechanical issues with a hard disk. The question is not IF a hard disk will fail, but rather WHEN. During my span of computing I've probably had around 40 hard disks. At least around 25-35% are dead by now. Some have had dead controllers, some have had the read/write heads hit the spinning platters, some of them stop spinning.

 

If you got important data, don't keep it in one place. And if you got important data, keep on moving it to new physical storage media every 2-3 years or so, depending on redundancy. The old storage media won't last.

[Bruno]

You can also try a Drobo, it's foolproof and simple to use, and you're supposed to keep replacing the smallest hard drives with bigger ones, and it takes care of everything for you. It never looked very professional to me, but it does look nice and simple.

[Sean Cunningham]

I've got four full-size ProMax towers with $16,000 worth (in 2002 dollars) of 10K RPM SCSI-160 platters that I'm half tempted to try to spin up now that I know the two DualMax RAIDs are still functional.  I just need to buy some earplugs first, lol.

 

What's sad is I've got a single half-height drive quietly spinning on my desk with the same amount of space as what I get with the ProMax array that I bought for $79.  It doesn't have quite the same throughput but it's really, really close.

 

All these fast SCSI systems were for the Cinewave-HD I build in 2002 (that also still works).  Uncompressed 422 used to be hard.

[dahlfors]

You can also try a Drobo, it's foolproof and simple to use, and you're supposed to keep replacing the smallest hard drives with bigger ones, and it takes care of everything for you. It never looked very professional to me, but it does look nice and simple.

 

Don't know if I'd call Drobo foolproof... http://scottkelby.com/2012/im-done-with-drobo/

 

Personally I'd rather go with Synology, QNap or Netgear, they seem far more reliable.

[nigelbb]

Unfortunately, I think thats your only solution.  :unsure:

 

I have a 3Tb hard drive that cost me £120. Nothing fancy but stores lots. Grab a few of them perhaps? 

 

 

or even better grab a few 3TB external drives at £85 each

 

http://www.ebuyer.com/434461-toshiba-3tb-stor-e-canvio-external-hard-drive-hdwc130ew3j1

 

[Brellivids]

 

 

There is no truly longterm backup solution for digital data.  This is one of the biggest issues for longterm film archival and one that the MPAA might have truly screwed the pooch on when that d-bag Jack Valenti was in charge.  Nothing has been invented that has a chance of outlasting archival film, properly stored.

 

 

edit: I can't imagine the hell it would be trying to find a working exabyte drive, or metrum, or the crummy software that was used to back data up back then.  Getting ahold of an old machine with a SCSI or SCSI-II interface and the right version of an OS.  That stuff was awful.  ZIP drives didn't last a year because the drives themselves were crud and they'd just start corrupting any disk you put in.  Is Iomega still around?  Oh look, they're Lenovo now.  I know never to buy any of their stuff, ever.

 

I watched this film that has been linked on these forums as well about the transition from film to digital. Key point there was that big studios pack a working reader + computer along the archival medias. 

 

I've had poor quality CD-Rs brake in two years (my friends first ever music production batch). In the days of CDR-R's I personally used high quality CD-R's for my storage needs and those discs do still work. How ever I would never use DVD or Blueray disks for archival purposes. Unless it is a third medium beeing used along say tape and HHD's