Editing and storage space

[Mozim]

Hey, 

 

I regularly shoot videos of Mountainbike events and my clients usually want to have the final result within a couple days, sometimes even hours after the event is finished. I'm currently editing on a mid-2013 MacBook Pro 15" Retina, 2.4 GhZ i7, 8GB memory and a 256GB SSD hard drive. 

 

The issue I have is related to the storage space... well lack of storage space that is. When I'm done shooting the event, I have between 50 and 200GB of video material from my cameras. Once that is converted to ProRes, the file sizes far exceed the storage space of the internal SSD drive. 

 

One (not exactly practical) option is to basically delete everything from my MBP before shooting the next event and that'll leave me with enough storage space to edit it on my laptop. The other option, and this is what I'm doing right now, is storing all the video footage on an external hard drive and editing straight off that hard drive. It's only a USB 3.0 connection so the connection speed is way too slow for real-time editing... which means that the editing process is quite frustrating. 

 

So, what's your suggestions to improve my workflow? Buying an external 500GB SSD drive might work but that'll be very expensive. Surely there must be an easier and cheaper option than that? 

 

Thanks a ton in advance, 

Moritz

[jonpais]

A 500GB SSD is NOT the solution to your problem. Are you going to continue deleting files to make room for new ones? Because you say you regularly shoot events that take up to 200GB on your drive, I am going to suggest something like the LaCie 5big Thunderbolt, a RAID with 10TB of storage with close to SSD speeds. If you are willing to wait, LaCie will be releasing a Thunderbolt 2 version with much faster speeds this quarter, as well as a couple of other Thunderbolt 2 models with read and write speeds of over 1,000 MB/s. I believe the 5big will run you about $1,000. There are many less expensive solutions, but they will be considerably slower, and turnaround time is important. Good storage does not come cheap. If you are looking for an inexpensive portable solution with read and write speeds over 200 MB/s, have a look at the Seagate Backup Plus Fast, a 4TB RAID that costs $270. I purchased one a few weeks back and am very happy with it.

 

/edit/Could you tell us what external drive you are using, and what the read/write speeds are?

 

/edit2/8GB RAM is insufficient for editing. Unfortunately, with the rMBP, you can't add more. 16GB should be considered the minimum. For example, with only 8GB RAM on a similarly specced late 2013 rMBP, If I have so much as one other app open at the same time I am editing in FCPX, I am working with virtual memory, which is going to hurt overall performance considerably. And you can forget about working in real time in DaVinci Resolve, especially without dedicated graphics. From what I've been reading here in the forums, far too many professionals are spending big bucks on gear but skimping out on their editing suite. One common reason for that is they are only taking into account present needs, without considering that in the future, they may be editing uncompressed files or 4K, using NLEs like DaVinci Resolve or using power-hungry plug-ins like Neat Video. Incidentally, there is no mid-2013 Macbook Pro: there is an early and late.

[Brellivids]

SSD prices have come down nicely lately.. At least in Euros.. 1000G 840EVO's were 500€ now they are about 370€

[jonpais]

@Brellivids That's a great deal on an awesome drive. Still, the OP needs a large capacity external drive, preferably more than one for secure backup. 1TB external storage is inadequate for video professionals, though I am seriously considering purchasing the soon-to-be released LaCie Little Big Disk Thunderbolt 2 1TB drive just for editing, not as a standalone storage solution. It contains 2 Samsung SSDs and achieves read speeds of over 1,300 MB/s, which will be way more than enough for editing the compressed 4K video from the GH4.

[Mozim]

Thanks for the advice!

 

Right now I'm using 2 Western Digital My Passport for Mac external drives. I use one to store the raw files (as in video raw material that I haven't edited yet) and the other one to store the projects and video material once the video is finished and exported. Read and write speeds are around 100MB/s. 

 

I bought the MacBook Pro in May 2013, so I assume it's the early 2013 one. It has a dedicated graphic card and the performance is fine for what I'm doing right now. Looking back at it now I could've gone with the 16GB RAM version but that would've been even more expensive and I just started making videos about a year ago, so when I bought the Laptop editing videos was far from being my main concern. Having to deliver fast and good results for clients within a short period of time started just a few weeks ago and I'm not a professional video maker. I know that I would like to buy a proper editing computer (not a portable one) soon depending on how things go though. Still, when editing smaller projects on my laptop the performance is great, despite the lack of RAM. 

 

Anyways... the way I work right now is I shoot the footage, usually between 50 and 200GB per event / weekend. This material is stored on my external hard drive #1. Then I go through the footage, make a pre-selection of shots and sequences etc. and then import the files into my editing suite (FCP X) where the files are converted to ProRes. The source files and the project are stored on the external drive because my laptop doesn't have enough storage space internally. Then I make various edits for various clients and they share some of the footage - this is why it's important for me to have one big project containing all the footage from the event instead of having various smaller projects. Once the edits are finished, exported and uploaded, I make sure clean up the footage (deleting scenes that I haven't used and won't use in the future, tagging the footage with keywords etc.). So now I'm done editing, and I won't need the projects or footage any longer unless a client asks for specific shots from the past. The last step is to transfer the entire project (raw material, converted material and final edits) from external hard drive #1 to external hard drive #2. Now I have plenty of space on hard drive #1 for the next event and I use hard drive #2 as a back up drive. Right now hard drive #2 is a portable hard drive but I'll buy a stationary external drive soon because that seems more secure to me. 

 

As stated earlier, the issue I have is that the performance of the system is fairly slow when editing straight off the external hard drive. This is NOT the case when editing straight off the internal SSD of my computer, but the SSD capacity is very limited and most projects are too big for the internal SSD. 

 

Once the footage is cut and exported, I don't really care how long it takes to access the project if I have to. An external drive with a ton of capacity and a USB 3.0 connection would be perfectly fine because it'll only be used to store old projects... maybe two instead of one to have a back up just in case. ;) 

 

The LaCie solutions look great but right now I can't really predict if I'll be making any money from shooting videos, so I'm a little hesitant to spend that much money. Do you think that the solution Brellivids suggested will work for me? Jonpais, have you tried editing off the Seagate Backup Plus Fast? Looks like a great solution if it's fast enough... 

 

Again, thank you so much for your help, I greatly appreciate it!

 

Cheers, 

Moritz

[jonpais]

The Samsung 840, as superb as it is, isn't the solution for you, since you can't replace the drive in your Mac (and even if you could, it's substantially slower than what you've already got in your machine). I haven't begun to edit the mini-doc I've been shooting yet, but I should begin by the middle of the week. The Seagate Backup Plus Fast is bus powered USB 3 and will reach speeds that are over double what you are experiencing now: you should see a significant decrease in transfer times. It should be adequate for editing ProRes 422, if not lightning quick. Unlike the slower (and considerably more costly) WD My Passport Pro (another portable RAID released at the same time) though, it is only configurable as RAID 0 (meaning, if one of the disks fails, you will lose all of your data), which rules it out as a sole storage solution. I think you really need to evaluate your storage needs. Continuing to buy cheap, low capacity drives will only end up costing more in the long run, with the added risk of losing data. I know: I used to own a WD My Passport for Mac. A single drop from a couple feet off the floor, and you will lose everything. Fortunately, I noticed it was clicking and was able to save the files to another drive before it died completely. Which is why I would normally not purchase a portable drive that wasn't shock resistant. And remember that a 4TB drive should really be thought of as 2TB, as performance drops as spinning drives fill up! By the way, it's not USB 3 that is the bottleneck in your system, it's the 5,400 RPM drives. The limit is around 120 MB/s for a single spinning disk, nowhere near saturating the USB 3 protocol. To sum up: if you must get storage now, the LaCie 5big 10TB has the capacity you are looking for, with read and write speeds comparable to the 840 (I'm not trying to suggest they are alike, however!). Think of it this way: if you wanted to buy 10TB of Samsung SSD 840s, it could cost over $4,000!

[Mozim]

Again, thanks for the input! For some reason I was assuming that the Samsung 840 is an external SSD drive that connects via Thunderbolt or USB 3.0 - and that would've been a fantastic solution, but I guess I'll have to look elsewhere. Of course I won't replace the internal hard drive, that's not going to solve any of my problems. 

 

Let me know about your experiences with the Seagate Backup Plus, it'll be very interesting to hear. RAID 0 means that it's one large 4GB drive instead of two 2GB ones that mirror each other, right? That's no big deal for me, since I'm more concerned about my stuff getting stolen instead of one of the drives actually failing. 

 

I'll also think about the LaCie 10TB drive but right now it's hard to justify the price since I'm just starting out. But you're right, I shouldn't just consider current needs but think about the future as well. 

[jonpais]

@Mozim Yep, the 840 EVO was designed for Ultrabooks - it is tiny as heck, reasonably priced, energy efficient and is the first mSATA 1TB SSD. The Seagate Backup Plus Fast actually has 2 Samsung 2TB drives. Even if it were possible to configure them like the WD, I wouldn't do it, because I'd be back to 100 MB/s speeds, which is not acceptable. For sheer performance, the Seagate rivals many desktop drives of similar capacity at a price that is unmatched. Another comparison: let's say you were insane enough to purchase 4 more WD portable drives; you'd pay $400, which is $130 more than the Seagate 4TB drive, and you'd be spending twice as much of your time waiting for your projects to render! Because theft, fire and floods do happen (and not just to other people), it is a very good idea to store drives in different locations, if at all possible. 

 

/edit/ I just noticed that you wrote Backup Plus. It is the Backup Plus Fast - huge difference!

[dahlfors]

If you can live with speeds around 80-100 MB/s, you could go for some of the faster NASes available (Synology / QNAP and similar ones). If you need higher speeds, Thunderbolt storage is the way.

 

Should you need larger storage in your MBP, it is possible to replace SSD, although it might not be something for everyone to do: http://www.ifixit.com/Guide/MacBook+Pro+15-Inch+Retina+Display+Early+2013+SSD+Replacement/17059

[jonpais]

I will see if I can't edit a little footage in FCPX this evening with the Seagate Backup Plus Fast attached to my 27" iMac. I'll try to let you know the results by tomorrow afternoon.

[jonpais]

I just did a short test of the Seagate Backup Plus Fast with FCPX on the iMac. First of all, read and write speeds as measured by the AJA Systems Test and Black Magic Disk Speed Test were both around 200 MB/s, results which by the way compare favorably with some popular 5-bay desktop storage systems costing several times more and weighing in at 15 pounds! Of course, those systems have many more niceties... Importing and transcoding 9GB of MOV files to ProRes 422 took seven minutes - which isn't going to win any speed records. I applied the 'aged film' look from special effects to one of the clips, but got tired of waiting for it to finish rendering, so I don't have any results to share with you. Now I can appreciate the work of those intrepid guys over at AnandTech, et al. The upshot is, unless you have the patience of Job, if you plan to spend a lot of time editing and color correcting, it pays to invest in storage with read/write speeds in the ballpark of 500MB/s. 

 

/edit/Just for the record, you don't have to wait for rendering to finish when working with FCPX, and in fact, I prefer to initiate rendering manually. And there are at least half a dozen tricks to get faster performance including reducing resolution in the viewer, and so on. And just doing simple edits is instantaneous.

[jonpais]

Speak of the devil! TweakTown just posted their review of the Buffalo DriveStation Mini Thunderbolt 512GB External SSD. In RAID 0, we see write speeds of 647 MB/s and read speeds of 717 MB/s. In RAID 1, the Buffalo reaches read speeds of 651 MB/s and write speeds of 320 MB/s - results very similar to the 2013 iMac 1TB Fusion Drive. This is some of the best performance we've seen from a first generation Thunderbolt external drive. MSRP is $650 and supposedly there are ample quantities in stock. I don't think I've ever worked on a project longer than 100GB, so this would be an ideal solution for me - it strikes a nice balance between size, speed and cost. Just to clarify - this is solely for editing 1080p or compressed 4K clips, not for long-term storage.

/edit/ While this drive costs a little more than Brellivid's 840 (which can't be installed in the MBP anyway), it offers the same capacity and much higher read/write speeds. It can also be daisy-chained.

[Brellivids]

One solution might be a third party external HHD raid box.

IT's best to remember that RAIDS are usually limited in read speed to the speed of the drives but they do write at twice the speed of one drives write speed since two drives are writing (RAID 0)

 

So external box that lets you connect SSD*s to via SATAIII it and has thunderbolt/USB3  would likely be the fastest and cheapest option to get fast external storage space.

 

I have no idea wheter such prodcuts are readily available atm or not.

[jonpais]

In fact, no single enclosure is satisfactory: a secure backup strategy is going to involve at least two or three backups, with one or more of them away from the computer. This is particularly true if you are concerned about theft or vandalism. Fault tolerant RAID in and of itself is not a backup. Here are some excellent resources for those wanting to learn more about storage, backup strategies, Thunderbolt and RAID:

 

http://www.tomshardware.com/forum/forum-32.html

http://macperformanceguide.com/index_topics.html - this is a must read 

http://www.larryjordan.biz/hard-drive-basics-a-collection/

 

"The speed of our computer is far, far less important than the speed and the size of our storage. And if I were to invest money, investing money in high speed storage is a far better investment than in getting the absolute fastest processor because any processor right now is sufficient for doing even 4K video." - Larry Jordan

[Axel]

Note: My following statement doesn't contradict anything written above, it's just a simple, old rule that's still useful:

 

If playback performance is paramount (and it generally is, who cares if after two days of editing the export takes half an hour or one hour?), the system with the applications should be on another harddrive as the footage. 

 

The HD with the footage should be 'read only', and it should be the fastest drive with the fastest connection, preferably an SSD.

 

That means if you allow the program to write render files during editing, these should go to another drive. You can also trigger rendering manually, in a pause. This way when you edit, the second drive is also 'read only'.

 

A backup drive can be a very slow drive. This can also be where you export your masters. 

[Caleb Genheimer]

I just got a DROBO 5D, and I love it. Fast (USB3), well-built and pretty much idiot-proof.

[jonpais]

Note: My following statement doesn't contradict anything written above, it's just a simple, old rule that's still useful:

 

If playback performance is paramount (and it generally is, who cares if after two days of editing the export takes half an hour or one hour?), the system with the applications should be on another harddrive as the footage. 

 

The HD with the footage should be 'read only', and it should be the fastest drive with the fastest connection, preferably an SSD.

 

That means if you allow the program to write render files during editing, these should go to another drive. You can also trigger rendering manually, in a pause. This way when you edit, the second drive is also 'read only'.

 

A backup drive can be a very slow drive. This can also be where you export your masters. 

Exactly. The footage should be on a different drive from the boot drive for maximum performance.

 

In FCP, if you go to System Preferences, then to the 'Playback' tab, you can choose to set rendering to begin after 60 seconds or even longer. If, on the other hand, rendering takes place in the default setting (in other words, far too often), you can always go to 'File' and select 'Delete Event Render Files' every so often to clear up space on your drive.

 

If you are looking for a 'read only' (i.e. just for editing video) drive, two Thunderbolt SSDs stand out from the rest: the Buffalo DriveStation Mini and the LaCie Little Big Disk Thunderbolt 2. The 512GB Buffalo scored 717 MB/s read and 647 MB/s write speeds according to the BM Disk Speed Test. The LaCie LBD 1TB has read speeds of 1,375 MB/s (but in first gen Thunderbolt, speeds will be around 800 MB/s). The former costs $650, the latter, $1,300.

 

For 'long-term' storage, the soon-to-be released LaCie 5big with Thunderbolt 2 and the OWC Thunderbay IV Thunderbolt are remarkable. The 5big starts at 10TB and promises transfer rates of up to 1,050 MB/s. No word on cost yet, but the current LaCie 5big 10TB with first gen Thunderbolt runs around $1,200 $1,000. Finally, there is Other World Computing's Thunderbay IV, which can achieve transfer rates of around 800 MB/s and retails for $870 with disks included. Both can be configured RAID 5, making them ideal for both editing and long-term storage, whereas the first two, because of cost and limited capacity, are obviously exclusively for editing.

 

Buying cheaper, slower, low-capacity drives just doesn't make any sense to me, and in fact, will end up costing more in the long run. 

 

How fast is fast enough? For the compressed 4K files from the GH4, transfer rates of ~400 MB/s are probably adequate. I won't have the camera in my hot little hands for at least a month to do any testing.

 

/edit/ Obviously (or maybe not!), there are many other factors than sheer speed involved in choosing an appropriate backup. I'm not a power user processing 100s of terabytes a week and running my drives 24/7; nor am I a working professional whose livelihood depends on having umpteen backups stored both on- and -off site; and I often do backups as often as several times a week anyhow; but having gone through the mill with inadequate computers, I don't want to end up throwing money out the window on slow, low-capacity drives, either. In any case, a top-of the line storage solution like the OWC costs no more than a premium Panasonic lens, and should provide years of reliable service.