Thumb/flash drive as longterm storage

[Jonesy Jones]

What do you guys think? What's the best affordable longterm solution out there right now?

I have been thinking about using a thumb/flash drive (or a bunch of them) as a possibility. They are not too expensive these days. Like this one. 

My thought at this point is not so much for storing all the source files and everything, but instead render out a few uncompressed versions (graded, ungraded, no transitions, etc) of each project and store those on the drive. I realize they don't last forever just sitting there, but my understanding is that if you fire them up now and then they could very well last forever. 

But what about the source files. I could live without them nearly always, and some projects could even fit on one of those drives above. 

What are your thoughts? Are there better affordable solutions out there?

[Inazuma]

I would consider using blu ray discs as the cost per GB is smaller and discs are immune to mechanical failure (unless theyre improperly stored and gather scratches)

[Don Kotlos]

Blue ray is your best bet. 

2.5" mechanical drives is the next best thing if you want to also save some space. 

Most thumb drives use very cheap materials so they are very prone to unrecoverable malfunction.  

*SSDs in this poster are SLC. The majority SSDs in the market are MLC/TLC which is much worse for the drive longevity 

[The Chris]

I'm using SD cards to archive projects these days. If I need to transfer them to anything in 10 or 15 years it's easier than bluray and more dependable than spinning drives. Costco has Sandisk cards really cheap.

Though I haven't revisited anything that I've archived, not once. I wonder how long I'm actually going to keep this stuff, stacks of drives or hundreds of bluray disks just isn't practical for me. 

[Jonesy Jones]

Blue ray is your best bet. 

2.5" mechanical drives is the next best thing if you want to also save some space. 

Most thumb drives use very cheap materials so they are very prone to unrecoverable malfunction.  

*SSDs in this poster are SLC. The majority SSDs in the market are MLC/TLC which is much worse for the drive longevity 

According to this graphic my idea of flash drives is actually the best. Though maybe not in typical thumb drive form but sd instead. That was actually my first instinct but just figured that the usb drive was convenient, but if they're not as good I'll just go with the sd version. 

[dahlfors]

SD / CF cards / USB memories:

Very resilient form of memory - there's lots of stories of cameras ending up soaked in water where the photos have been able to be read once the cards have dried up. The one weak part is the storage controller chip. If the controller dies - you can't access the memory by yourself unless you know your way around electronics really well and can hook up a 3rd party memory controller. Rarely the memory itself goes bad unless there's serious damage (like a fire) - so data rescue companies will likely be able to rescue data - for a price.

 

Blu-ray:

I've had properly stored DVD-R's and CD-R's have their data layers falling off for no reason after less than 10 years of storage (some of these failed ones were even expensive high-end brands marketed for their durability for long-term storage - while the cheapo brands have fared just as well). Most of the disks stored in the same manner for the same time period have had no issues at all. Due to this kind of random failure I think you'll never know beforehand which brands are safe to buy - or which batches from a brand that are ok. Hence I wouldn't recommend Blu-ray discs unless you do two copies every time. BD-R's on the other hand are very cheap - but if you got large projects you'll need to split them up on several discs.

 

SSD:

SSD's are actually same price or in general cheaper in price/GB than SD cards or USB memories. Very resilient memory - same thing applies as for other types of flash memory, the storage controller is the part that can fail. Even if storage controller fails, data should be possible to recover by data recovery companies - for a price.

 

Conventional hard disks:

The price/GB is still many times cheaper for hard disks than most other medias (BD-R's beat them, some tapes for tape drives might too). Due to hard disks often failing, and due to their fairly low price/GB - I'd recommend a NAS solution with proper redundancy. Redundancy means - should a hard disk in a NAS break, you don't lose data and don't need to find some means of recovery. You just insert a new hard disk and let the disk array rebuild. I'd recommend a NAS that has RAID-6 (2-disk redundancy) and at least 5 hard disks for longterm storage. The initial cost will be bigger, but for larger amounts of data and large projects this solution will win considered storage/price. Conventional large capacity hard disks will likely get ever cheaper the coming years due to larger & cheaper SSD's, until hard disks finally go extinct as a storage medium - in about 5-10 years.

 

Conclusions:

- If your data amounts that you want to put away for long-term storage is small - say, 500 GB per year, some flash-based memory will probably do fine. If your needs are larger than 4-5TB per year or so, I'd definitely recommend a NAS-based solution.

- If you know your projects will fit inside the size limit of BD-R's so you don't need to split discs - they're a very cheap solution, even if you burn double copies of the project for safe storage.

[Don Kotlos]

According to this graphic my idea of flash drives is actually the best. Though maybe not in typical thumb drive form but sd instead. That was actually my first instinct but just figured that the usb drive was convenient, but if they're not as good I'll just go with the sd version. 

Yes some SDs might be good enough but the majority are using cheap flash which loses voltage over time and might result in data loss. 

[Kristoferman]

Consider Amazon cloud storage. Unlimited cloud storage for like a hundred bucks a year.

[Jonesy Jones]

SSD's are actually same price or in general cheaper in price/GB than SD cards or USB memories. Very resilient memory - same thing applies as for other types of flash memory, the storage controller is the part that can fail. Even if storage controller fails, data should be possible to recover by data recovery companies - for a price.

...

The price/GB is still many times cheaper for hard disks than most other medias (BD-R's beat them, some tapes for tape drives might too). Due to hard disks often failing, and due to their fairly low price/GB - I'd recommend a NAS solution with proper redundancy. Redundancy means - should a hard disk in a NAS break, you don't lose data and don't need to find some means of recovery. You just insert a new hard disk and let the disk array rebuild. I'd recommend a NAS that has RAID-6 (2-disk redundancy) and at least 5 hard disks for longterm storage. The initial cost will be bigger, but for larger amounts of data and large projects this solution will win considered storage/price. Conventional large capacity hard disks will likely get ever cheaper the coming years due to larger & cheaper SSD's, until hard disks finally go extinct as a storage medium - in about 5-10 years.

I hadn't done the math on the SSD's yet so that's a good thing to look into. Good to know about NAS. Will have to research that a bit though.

Consider Amazon cloud storage. Unlimited cloud storage for like a hundred bucks a year.

I hadn't thought about this seriously at all, nor had I heard about Amazon's service. I will look into that now. What happens to all your currently uploaded data when you stop paying the yearly premium?

[dhessel]

absolutely not, have had many fail all of a sudden with out notice. Either use cloud storage or a mirroring raid, IMHO.

[Kristoferman]

I hadn't done the math on the SSD's yet so that's a good thing to look into. Good to know about NAS. Will have to research that a bit though.

I hadn't thought about this seriously at all, nor had I heard about Amazon's service. I will look into that now. What happens to all your currently uploaded data when you stop paying the yearly premium?

I was wrong it's only 60 bucks a year hah. 

I'm sure you wouldn't be able access your files and they reserve the right to delete them if you don't pay. I'd just recommend setting up the automatic payments and you're good to go. It's kind of an unbelievable deal imo.

[Jonesy Jones]

I was wrong it's only 60 bucks a year hah. 

I'm sure you wouldn't be able access your files and they reserve the right to delete them if you don't pay. I'd just recommend setting up the automatic payments and you're good to go. It's kind of an unbelievable deal imo.

Can you paste the link here? I've looked it up and can't seem to find the deal you are referring to. Maybe I'm looking at the wrong service.

Edit: never mind found it. I was looking at the wrong service. https://www.amazon.com/clouddrive/home

[Kristoferman]

Yep that's it! 

[Inazuma]

Seems to be just US only   even so i don't think it's without risks. I mean they basically end up holding your data hostage for money. What if they bump up the price every year and then one year you've really hit rock bottom and can't make the payment. Then your stuff is gone if you don't have a backup somewhere else. 

Or another scenario. Someone gets your login credentials and starts fucking around with your files because he's the joker and just wants to watch the world burn

[Kristoferman]

Seems to be just US only   even so i don't think it's without risks. I mean they basically end up holding your data hostage for money. What if they bump up the price every year and then one year you've really hit rock bottom and can't make the payment. Then your stuff is gone if you don't have a backup somewhere else. 

Or another scenario. Someone gets your login credentials and starts fucking around with your files because he's the joker and just wants to watch the world burn

Well that's why you have a legitimate password. And if you think you wont be able to pay you download all of your files and back them up some other way (which would probably cost significantly more than whatever the yearly fee is). I dunno, unless you're enemies with mr. Robot , I don't see it as a bigger risk than backing it up yourself. 

[Jonesy Jones]

Just saw this. It's for aws but looks amazing. Like unbelievable actually  

https://aws.amazon.com/importexport/?tag=nofilmschool-20

[Xavier Plagaro Mussard]

The only safe long-term storage would include several methods (SD, bluray, hard disk, usb stick, cloud) in several physical locations. 

[sudopera]

Does anyone have some experience with these, it seems like a good solution for archive/backups and not too expensive

HP

http://www.bhphotovideo.com/c/product/1044076-REG/hp_c8s07a_rdx_usb_3_0.html

http://www.bhphotovideo.com/c/product/1044079-REG/hp_q2042a_rdx_cartridge_500gb.html

Tandberg

http://www.bhphotovideo.com/c/product/1111616-REG/tandberg_data_8781_rdx_quikstor_usb_3_0.html/prm/alsVwDtl

http://www.bhphotovideo.com/c/product/1070103-REG/tandberg_data_8541_rdx_rdx_quikstor_500gb_remvble.html/pageID/accessory

[IronFilm]

Could should only be one part of your back up solution. 
Ditto storing "back ups" in your closet isn't enough. You need a back up stored off site somewhere as well, for some people going for cloud storage could be a simple answer to this rather than driving 10hrs to your grandmother's to drop off a harddrive every time you do a fresh back up. (or whatever your current offsite solution is)

[blafarm]

I wouldn't depend on the accuracy of that "Lifespan of Storage Media" chart.  

30 years for DVD-R?  That's completely moronic, the organic dye layer is absolutely capable of failing in less than 5 years.

And 34 years for a Hard Drive?  Please tell me exactly what drugs this person on?  Nobody's hard drive lasts that long. 

If you follow the enterprise players, you quickly understand that their approach to archival storage is LTO -- which isn't even on that chart (another epic failure).

However,  there are other options that are worth noting.  This is a piece I put together for a client.  It is not a promotional piece and I was not paid to write it.  You might find it interesting.

M-DISC is a mineral-based optical disc that holds up to 100GB of data and has a reported shelf life of 1,000 years.  The principal difference between this product and the current crop of optical media is that these discs do not use organic dyes that fade and decay.  Instead, M-DISCs use a data layer consisting of chemically stable, heat resistant metals and inorganic materials that are reported to last centuries.  In other words, data is ‘engraved’ into the M-DISC by physically altering the recording layer and creating permanent surfaces in the disc.

The US Department of Defense (DoD) tested five different brands of archival-quality, gold dye-based recordable DVD discs and the M-DISC DVD.  The tests were conducted to determine which media would hold-up under the harshest environments over long periods of time.  None of the M-DISC media suffered any data degradation at all.   Every other brand tested showed large increases in data errors and many of the discs were so damaged that they could not be recognized as DVDs by the disc analyzer.

M-DISCs have been around for a while, although they have previously only been available in DVD (4.7GB Single Layer) and BD (25GB Single Layer) sizes.  Recently, M-DISC released 100GB BDXL discs and the data densities are finally beginning to make sense for larger archives. 

BDXL discs are the successor to the original single and double layer Bluray Discs, which have capacities of 25GB and 50GB respectively.  BDXL discs are designed to hold 100GB and 128GB and are principally targeted for distribution of 4k/UHD media.

M-DISCs can be burned in a very wide variety of commodity-priced, consumer-level optical drives, and can also be burned using ultra-slim optical drive form-factor drives that are integrated into laptops.  Not surprisingly, optical drive manufacturers have gotten behind this format in a big way, most notably Hitachi and LG. 

In fact, if you happen to have a more recent DVD or BD burner in your computer or facility, take a look at the front panel to see if the M-DISC logo is there.  The M-DISC logo on the front of the drive means you can burn M-DISCs that are supported by that particular drive (DVD, BD, BDXL).

But the more important part of this equation is that M-DISCs are ‘backward compatible’ and can be read in any other optical drive.  In other words, while you need an M-DISC-compatible drive to ‘write’ or ‘burn’ a disc – any other consumer drive can ‘read’ them.  That means your archives can be read in any other optical drive in your facility, or your client’s facility – and the data is random access.

In terms of pricing, when purchased in 50-Pack quantity on the M-DISC website, the DVD unit cost is $2.78 each, while the BD unit cost is $4.40 each.  Note that the recent release of the larger 100GB BDXL product means that supplies are limited, and 15/50-Pack quantity pricing is not yet available.  More specifically, the largest size of BDXL discs is currently limited to a 5-Pack and the unit cost in that quantity is $19.90 each.  This represents a small price premium compared to purchasing 4-25GB BD discs having the same storage capacity for a total cost of $17.60.

Obviously, on a comparative basis, you would need 25 -100GB M-DISCs at a total cost of $497.50 to equal a single native 2.5TB LTO6 tape which is typically priced at ~$35.  As such, on face value, a direct comparison to LTO6 appears quite unfavorable.  However, M-DISC is clearly not an archival solution that is meant to compete directly with LTO6, unless of course you have a lot of money and your archive sizes dovetail nicely with M-DISC sizes.  

Another factor to consider is that LTO hardware and software can easily cost $5-10,000.  And that number doesn’t include potential annual LTO software maintenance contracts and the likely necessity of dedicating a higher-end computing platform to a tethered LTO drive, although that obviously doesn’t apply to a network-enabled LTO drive.

Conservatively speaking, at the very bottom end of that price range, and with the current limited quantity pricing for the BDXL product, $5,000 saved equates to being able to purchase 251 -100GB M-DISCs with a total storage capacity of 25TB.  And if you're facing a total cost of $10,000 for your entire LTO solution, it’s obviously double that.  But again, M-DISC size limitations have to be acceptable for your workflow, including the lack of cataloging.

Frankly, in addition to project archiving, there are many other uses for M-DISC, including show masters & protections, archiving sound effects and stock music libraries, DI and final mix elements, photos and client artwork, as well as accounting and tax records – all of which are frequently stored on spinning media in my world.  For those purposes, DVD and BD versions of M-DISCs are less expensive and more closely match the size requirements of those files.

M-DISCs can also be used for file-based deliverables that need to be shelf-stable for many, many years, albeit within the 100GB capacity constraint.  Your clients and vendors already have DVD and BD drives, all of which are automatically capable of reading M-DISCs with no problem.  And clients who would need the larger 100GB storage capacity of BDXL discs wouldn’t blink at spending ~$50 price for a new BDXL drive.  That’s a petty cash expenditure, and the BDXL drive they purchase doesn’t even have to be an M-DISC-compatible drive (not that they cost any more) if all they need to do is read the discs.

The broad 1,000 year claim is certainly a potential point of concern.  However, the DoD tests linked below instill quite a bit of confidence and Hitachi has incorporated M-DISC into their Digital Preservation Platform (DPP) which is an archival solution marketed to federal governmental agencies.  An M-DISC lifetime study conducted according to the ISO/IEC 10995 Standard indicated that the expected mean life was 1,332 years and that the expected 5% failure time was 667 years.

For me, the more important issue is that I don’t need 1,000 years of longevity.  At most, I need 20-30 years.  And, frankly, I really only need a very reliable 5-10 years, at which point the fast-paced world of technology will likely deliver a ‘next gen’ storage solution that replaces LTO and M-DISC.  For the near term, M-DISC offers a solution that avoids the inevitable failure of solid state and mechanic storage, as well as the data rot of organic dye-based optical discs.

If you can deal with the pricing and capacity issues, the arguments for this technology include:

 

 - A slew of commodity-priced drive options that can burn M-Discs (~$50)

- Discs can be read in any other optical drive -- on any other computer

- Interface options include:  USB, Firewire, Thunderbolt, eSATA and SATA

- Burners can be easily swapped between different computing and OS platforms

- No need for HBA (Host Bus Adapter) or proprietary hardware/cabling

- No need for a dedicated high-end host computer (e.g.: LTO)

- No need for proprietary drivers, middleware, software or catalog migration

- No risk of a LTO software vendor ‘falling down’ and their product becoming EOL

- Random access to folders and files with no ‘hunting’

- Compatibility and interoperability with clients and vendors (regardless of their technical sophistication)

- Media has no moving parts and the read mechanism has no physical contact with media

- Media is resistant to temperature, light, humidity, water, RF, EMI, magnetic and mechanical failure

- Media does not need to be ‘exercised’ several times a year (e.g.: hard drives)

- Existing ecosystem of optical media cases, mailers, shippers and labeling options

- Small and robust form factor helps with shipping to clients, vendors and offsite storage locations

For those not using a LTO-based solution for critically important data-at-rest, and who are rolling-the-dice relying on individual hard drives for that purpose, M-DISC would at least add a layer of backstopping.  The same might hold true for those using NAS boxes running RAID5/6. 

And for offsite archives (which really should be considered a mandatory requirement), M-DISCs offer a nice form-factor that can be put in a standard cardboard mailing shipper or UPS/FedEx envelope, and they are resistant to the physical, magnetic and environmental issues associated with shipping and offsite storage, which are often times beyond our control.

M-DISC is clearly not for everyone, but I like the idea of being able to restore data using any optical drive on any computer, and not being reliant on 3rd party software (or the software company’s ability to survive in the market).

I’m not suggesting for a moment that this technology replaces LTO, or the sophisticated software products that support it.  However, there are a number of use-cases for which this product might offer a viable solution.

Here are some links:

M-DISC Home Page:

http://www.mdisc.com/

M-DISC Store (products are also available on Amazon)

http://store.mdisc.com/

M-DISC Technology:

http://www.mdisc.com/mdisc-technology/

M-DISC Test Data:

http://www.mdisc.com/proving-ground/

M-DISC Drive Compatibility:

http://www.mdisc.com/m-ready/

Department of Defense Naval Air Warfare Test:

http://www.esystor.com/images/China_Lake_Full_Report.pdf

More specifically, see Document pages 43 and 44  (PDF pages 48 and 49)

Wikipedia BDXL Disc:

http://en.wikipedia.org/wiki/Blu-ray_Disc#BDXL

BDXL Drive Example:

http://www.amazon.com/LG-Electronics-Internal-Rewriter-WH14NS40/dp/B007YWMCA8