iMac or specialist editing rig?

[j.f.r.]

On August 18, 2016 at 11:45 AM, The Chris said:

Apple quietly made a few upgrades to the Late 2015 iMac retina update, most notably the screen is now 10-bit and the SSD's are PCI-e 3.0 - which makes them ridiculously fast. I tested one side by side with my 6-core mac pro and the iMac smoked it. So I sold the trashcan and bought a loaded iMac, 1tb SSD, 32gb ram and the 395x card. How fast? Check this out.

 

 

Here I am thinking my external SSD around 450MB is extremely fast..... Thank you sir for lettming me know I am living in ancient times.......

[Axel]

1 hour ago, j.f.r. said:

 

Here I am thinking my external SSD around 450MB is extremely fast..... Thank you sir for lettming me know I am living in ancient times.......

Right. But unless you plan to PiP multiple 4k ProRes4444XQ clips (or 4k raw, but FCP X can't handle that anyway), you won't notice the difference. You would, however, realize that the 1 TB drive in this example was way too small and you had to go external from then on. 

Another question. I used to have a three-monitor setup with my ancient MP. With 5k, everything fits on one screen. Some day I know I will want an external monitor, but all 4k monitors on the market in a justifiable price range don't seem to be on par with the excellent colors of the iMac display. What to do? 

Would an external 4k display slow down GPU performance significantly?

[The Chris]

4 hours ago, Axel said:

Right. But unless you plan to PiP multiple 4k ProRes4444XQ clips (or 4k raw, but FCP X can't handle that anyway), you won't notice the difference. You would, however, realize that the 1 TB drive in this example was way too small and you had to go external from then on. 

I can only speak from my experience, I notice a significant difference in speed from my previous 6-core trashcan mac pro, and I'm not editing PIP multiple HQ444 clips or raw anything. Everything is faster, with an hour of footage on the timeline comprised of a few hundred clips, transitions, color correction and so on I can scrub through everything with no hangups, transcoding to prores is faster, exporting is faster, etc. With the trashcan I only had a 256gb SSD, so I was editing off an external drive, the whole process was slower and noisier thanks to spinning drives.

My drive may be way too small for your needs, but its great for what I do, but finished pieces for me are rarely longer than 20 minutes, most are in the 3-5 minute range. I have over 800gb in free space, plenty of room for hours of 4k footage or a few hundred thousand photos. I edit on the SSD, move finished projects to external storage and clear everything off the SSD for the next project, so I can take advantage of the speed - which (along with the screen) is the whole point of me dumping the Mac pro for an iMac.

As always, YMMV.

6 hours ago, j.f.r. said:

 

Here I am thinking my external SSD around 450MB is extremely fast..... Thank you sir for lettming me know I am living in ancient times.......

That PCIe 3.0 really raised the bar. I was astounded at those numbers too since I was getting around 700MB from the trashcan, which itself was fast.

[M Carter]

On August 19, 2016 at 1:25 PM, Axel said:

I wouldn't. With 4k, you need more storage. My Pegasus is no bottleneck, though it only reaches 500 MB/s read speed. I've seen a test with 8, 16, 24 and 32 GB RAM on the maxed-out iMac. Didn't make much of a difference with FCP X. Slightly faster with 16GB, the sweet spot. FCP X is not as ram-hungry as Adobe. CPU and GPU are crucial. You can set free a lot of CPU performance with optimized media. Since those eat more space, you need it external. Big and fast enough TB drives are the way to go. With redundant arrays you need no extra backups.

I am finding drive speed to be less important for FCPX. But After Effects still seems to need the basic Adobe recommendation - app on a fast boot drive, scratch disk on the fastest shizz ya got, footage and content on a fast drive. As I migrated from the tower to the cylinder, I tested lots of stuff, and that's still the way to go with AE.

My first step for any gig is "everything gets pro-res'd". Since shooting 4K, I've found a 4TB RAID 0 (and a 4TB single-drive backup that runs at night) has been workable. Biggest issue is the size that job folders can become. Archiving a terrabyte at a time isn't unusual for me, and that takes 6-8 hours to transfer the content. More and more, I'm finding managing my archives and backups takes more planning as file sizes go up.

 

[Axel]

39 minutes ago, M Carter said:

More and more, I'm finding managing my archives and backups takes more planning as file sizes go up.

Did you try the Library Manager? 

Reading this thread, I realize I made a mistake with my current system. I bought the 256 flash version because that's enough for the system (and crazy fast too) and because I thought, well, people are doing this with the cylinder, it must work.

But now I learn that the iMac "smokes" the nMP. I should have ordered a bigger drive and use the Pegasus to store finished projects. That's the disadvantage of a compact system: I can't change the drive now, I have to live with it for at least two years. However, it's no tragedy, I have not been editing multicam for quite a while, and I don't feel any limitations so far.

As for AAE, it told me something like poor graphic card acceleration with your system on first startup. I used to have Nvidia before. Doesn't bother me much. AAE never was a real time app for me

[joema]

On 8/19/2016 at 1:25 PM, Axel said:

...With 4k, you need more storage. My Pegasus is no bottleneck, though it only reaches 500 MB/s read speed....You can set free a lot of CPU performance with optimized media. Since those eat more space, you need it external. Big and fast enough TB drives are the way to go. With redundant arrays you need no extra backups.

I agree with all of these points except the last. I have a Pegasus R4 and several other larger Thunderbolt RAID-5 arrays on a top-spec 2015 iMac 27. 

The OP mentioned editing photos via Lightroom or video via Final Cut or maybe Premiere Pro, 1080p footage and later editing 4k. Those have entirely different demands. Almost anything can edit H264 1080p. By contrast significant amounts of H264 4k is really hard on almost any computer. FCPX is considerably faster than Premiere CC (I have both) but even FCPX can bog down on 4k, esp. multicam. It generally requires transcoding to proxy for smoothest performance on 4k, which takes lots of space. Proxy is about 2x the space of H264 camera files and optimized ProRes is about 8x the space.

I have six Mac and one Windows machine and like OS X but if I were editing mostly on Premiere I'd build or buy a high-end Windows machine for this. You have a lot more configuration options and (as of today) the performance options on the Mac side are limited. This will probably change this fall with the new iMacs and hopefully refreshed nMP.

Re Lightroom, it can definitely be sluggish even on a top-spec iMac 27 if editing lots of high-megapixel raw stills. It is unclear if this is a GPU limit due to the 5k screen or a CPU limit (say from bit-block-transfer operations). If you do lots of production work, e.g, an event photographer shooting > 1,000 38-50 megapixel raw stills a day, a high-end Windows machine is probably better.

Your "fast enough" statement is correct and often misunderstood. For most video editing it generally doesn't help to have 1,000 or higher MB/sec -- often obtained at great cost financially and sacrificing larger size. Long before you need 1,000 MB/sec you are bottlenecked on CPU or GPU.

And as you said, having media on smaller super-fast storage means you often don't have space to transcode to more efficient codecs. This means the high-speed storage has actually made the performance problem *worse* not better, since the most common limits are CPU and GPU not I/O.

However I don't agree redundant arrays eliminate the need for extra backups. You can easily have a problem from user error, system software error, application software error, etc. which jeopardizes your data. RAID only helps for disk hardware problems. At a bare minimum I'd suggest Time Machine backup and it's really best to have a disconnected off line backup using Carbon Copy, etc. in addition to Time Machine. And for critical material you probably want additional backups beyond these.

[red]

Thank you so much to everyone who has posted so far. Lots to read up on today and then evaluate and ask more questions

Great community as always!

[joema]

Re the point Axel and I made about importance of CPU and GPU and limited importance of I/O beyond about 500 MB/sec on H264 video editing, I should add that some highly-experienced people feel otherwise. Larry Jordan for example says I/O is the most important, GPU next and CPU last, and that you can edit 4k on almost any computer. That is roughly the opposite of my experience as a professional video editor using both FCPX and Premiere CC, but I edit a lot of H264 and only transcode to ProRes or other lower-compression codecs when it's unavoidable. In general I/O rates aren't that high when editing a long-GOP codec because otherwise the puny little CPU in the camera could not write the data to the card fast enough.

When configuring a computer for editing,  I/O is important but buying more I/O than is necessary usually results in short-changing yourself elsewhere. E.g, getting an external SSD array then running out of space because you didn't realize how rapidly video editing consumes disk space.

[M Carter]

On August 22, 2016 at 4:24 AM, Axel said:

Did you try the Library Manager? 

That's not the issue for me as far as backups and archiving. Mainly making sure I have a 1 and 2TB raw drive on hand when my RAID fills up, being a little more aggressive about trashing things that don't need to exist (I archive to raw drives with a USB 3 dock). I don't deliver or edit 4K, but I do a lot of 4K on a 1080 timeline, so it's not unusual for a longer-term project folder to hit 1TB... and that's about 6 hours to copy to an archive, test the archive, and delete from the RAID and the backup. So I try not to let archiving backlog as much as I used to. Then again, nothing sucks like archiving and then the client calls and says "so and so's title just changed, can you do a new edit?" Luckily, I use a dock and raw drives so I can do quick things right off the archive, but I try not to use those discs much.

On August 22, 2016 at 4:24 AM, Axel said:

Reading this thread, I realize I made a mistake with my current system. I bought the 256 flash version because that's enough for the system (and crazy fast too) and because I thought, well, people are doing this with the cylinder, it must work.

I did the same. I just bought one of these since I came from a Pro Tower internal RAID. Seamless and reasonably fast (300mb/s or so), $300. You can get a USB 3 RAID box for dual SSDs even cheaper, though drive costs are an issue. I have 7200 RPM spinners in this, 2TB each for a total 4TB working space. I have a 4TB drive in a USB 3 enclosure for backup at night. It's a great product and far better reviewed than the OSC/MacSales raid. Quiet, too. I have no issues with FCPX or After Effects (though I use an SSD for AE scratch files).

I've never wanted anything but my apps, email, and billing on the internal drive. And I still think the Cylinder is a fail as far as no drive bays and relying on USB and Tbolt (and the iMac is no answer to that). But Apple always knows better than us, right?

 

[Axel]

9 hours ago, M Carter said:

I archive to raw drives with a USB 3 dock.

I consider most original footage (aside that from BM cameras, ProResHQ or Raw) to be "small". For archiving, I use FCP X camera archives, which back-up the whole cards prior to import. These files, though they look proprietary with their elsewhere unknown padlock-icon, show all camera folders on a Windows system, just one extra "something.fcarch" file. With FCP X, there is one big advantage over finder-backups: all archives that are connected to your Mac whilst FCP X starts up are indexed and recognized, no matter where you stored them or moved them. That means as long as you know these archives exist, you can trash any copies, and FCP X will spontaneously make a new copy if you want to see the clip. Footage clips can't go offline then - re-imports from AAE can, and that's where I feel the difference. I need to manually create folders and subfolders (and, if the project takes longer, backups, it may be careless, but I only timemachine my system).

Even with a maxed-out nMP I believe I would work with optimized media. Faster machines may make H.264 4k editing smoother, but I always want the smoothest editing experience. For that I put up with an hour or two transcoding my favorites to ProRes "in the foreground". BTW, that's the only time I noticed my iMac had a fan that made noise. I don't hear it during export, because I toggle on background-rendering when my images are locked and I start fiddling with audio niceties, and therefore export times are just a few minutes with no rendering involved.

It may well be that ProRes typically takes 8 times the disk space compared to original media. As my Library-Manager screenshot shows, this ratio is much more reasonable if you don't transcode everything but just the usable stuff (it's german, but I think you will understand):

Everything you see here is a copy already. With a camera archive, you can't leave files in place, you have to copy them. That means I can unconcernedly trash them all once the project is finished.

Recently I discussed this with my Windows buddy, who finds the option to consolidate projects with referenced H.264 originals and Premieres smart rendering to be superior. True is, in the example above he would end up with, say, 60 GB of disk space to archive the project compared to 233 GB with FCP X camera archives ...

[M Carter]

On August 25, 2016 at 3:03 AM, Axel said:

I consider most original footage (aside that from BM cameras, ProResHQ or Raw) to be "small". For archiving, I use FCP X camera archives, which back-up the whole cards prior to import.

I think you're misreading what I mean by "Archiving". Sure, original card is in there, and the edit files. And the 1080 and 4K prores versions of everything, the after effects files and renders, the stills, the audio, the protools folder with audio sweetens or repairs or edits or music, all the client notes, storyboards, and edits, the bid, the revises to the bid, the script and revisions, reference images, edits from first pass to final, and Flash or Carrara animations and their associated image sequence folders... I mean, the whole freakin' job, with the entire folder structure preserved. 

100-200GB folders are the norm, but I just archived a 1.2 terrabyte job (6 or 7 shoot days, primarily interviews and b-roll, and tons of graphics). I had several brand new 1TB drives, tried like hell to prune that folder down. I generally archive when my 4TB RAID gets over half full, if possible. As I mentioned earlier, this can be 5-10 hours of copying and file managing. Doesn't shut the computer down, just makes it slower, so I try to do it overnight or on a weekend... except I seem to work a lot of weekends...

I just convert everything to ProRes from the start though, render everything from AE as prores (unless I need an alpha channel) and generally do image sequences from other programs and convert those to prores if needed. It takes very little time (I usually do it while I'm checking cameras and gear back in) but saves me a lot of hassle down the road. Disk space is cheap.

 

[Mat Mayer]

Visited an electronics store the other day and saw a 27 inch iMac which should be just fine for £1500, slow rendering but I can live with that if it means not paying double for the top spec versoon I had convinced myself I would require. Will just add an extra 16GB Ram for a small amount.

Choosing on the website I cant help but pick the most expensive option but Macs are very efficient so the 1.5k option should be fine, and an absolute joy to use (dedicated 2gb graphics, i5, 8gb RAM, 1TB HDD).  I just cant get used to using Windows, using an iMac makes all work fun.

[Axel]

@M Carter

Amazing. Of course I am only a wedding videographer who practically never needs to touch an old project. Part of the marriages go sour before I would feel the urge to free disk space. Lucky me.

8 hours ago, M Carter said:

I generally archive when my 4TB RAID gets over half full, if possible.

I wonder why you then don't buy a Pegasus R6 or R8. They are getting faster the bigger they are.

@Mat Mayer

Agreed.

[The Chris]

9 hours ago, M Carter said:

I think you're misreading what I mean by "Archiving". Sure, original card is in there, and the edit files. And the 1080 and 4K prores versions of everything, the after effects files and renders, the stills, the audio, the protools folder with audio sweetens or repairs or edits or music, all the client notes, storyboards, and edits, the bid, the revises to the bid, the script and revisions, reference images, edits from first pass to final, and Flash or Carrara animations and their associated image sequence folders... I mean, the whole freakin' job, with the entire folder structure preserved. 

100-200GB folders are the norm, but I just archived a 1.2 terrabyte job (6 or 7 shoot days, primarily interviews and b-roll, and tons of graphics). I had several brand new 1TB drives, tried like hell to prune that folder down. I generally archive when my 4TB RAID gets over half full, if possible. As I mentioned earlier, this can be 5-10 hours of copying and file managing. Doesn't shut the computer down, just makes it slower, so I try to do it overnight or on a weekend... except I seem to work a lot of weekends...

I just convert everything to ProRes from the start though, render everything from AE as prores (unless I need an alpha channel) and generally do image sequences from other programs and convert those to prores if needed. It takes very little time (I usually do it while I'm checking cameras and gear back in) but saves me a lot of hassle down the road. Disk space is cheap.

 

Geez, that's a lot of stuff to store. Most of my projects will fit on a 64 or 128gb flash drive - which is my preferred archival medium. I'm not into endlessly replacing spinning platters, but much like Axel, I really don't have many pieces I have to keep forever. I have a bunch of stock footage type b-roll, but little else. I do have a safe deposit box with a growing number of thumb drives though.

I've read some photographers say they never delete any raw file. I really don't see the point as I'm always generating enough new content that I'll never re-edit old files. One I finish an edit I'm happy with, it goes to the cloud, into a finished folder on my external array and the current archival drive. Some get printed, the rest are forgotten in folders on my drives that I rarely go back to. Seems silly at times. It a viscous cycle. 

[M Carter]

On August 26, 2016 at 5:59 PM, The Chris said:

Geez, that's a lot of stuff to store. Most of my projects will fit on a 64 or 128gb flash drive - which is my preferred archival medium. I'm not into endlessly replacing spinning platters, but much like Axel, I really don't have many pieces I have to keep forever. I have a bunch of stock footage type b-roll, but little else. I do have a safe deposit box with a growing number of thumb drives though.

I've read some photographers say they never delete any raw file. I really don't see the point as I'm always generating enough new content that I'll never re-edit old files. One I finish an edit I'm happy with, it goes to the cloud, into a finished folder on my external array and the current archival drive. Some get printed, the rest are forgotten in folders on my drives that I rarely go back to. Seems silly at times. It a viscous cycle. 

Maybe so, but I do a lot of B2b corporate stuff, which may be half-hour interviews cut to a minute or two, motion graphics, animation, etc. I do a lot of animation (which I use mostly AE for) so there are folders full of image bits and pieces, storyboards. Average final gig folder is anywhere from 60MB to 200MB, and I do need to keep every last little bit, and keep the folder structure preserved. ProRes does keep my overall projects size a good deal larger, but the benefits outweigh that in my case, and it would really take something amazing for me to drop ProRes as my default across the board.

I also do some master-planned community stuff, but these are sometimes 10-year to 15-year sales spans over hundreds of acres, and we pick up stuff again and again. So keeping track of things is critical, and I need to get to 'em fast sometimes. So spinning drives and a dock are optimal, with text or excel files for content lists. So I can search a folder full of text files (drive1.txt, drive2.txt and so on - I'm up to drive 18 now I think) to find the file with the disc contents and grab the drive.

On August 26, 2016 at 5:07 PM, Axel said:

I wonder why you then don't buy a Pegasus R6 or R8. They are getting faster the bigger they are.

@Mat Mayer

Agreed.

I don't like the idea of all that archived work being powered-up and live. Since all discs eventually die, I try to keep them sealed in drive boxes and not run them unless needed. if I had a pile of big drives always spinning, that could be 10TB or something... and then I'd need 10TB of constant backup. 4TB is a good size for me right now, a 2x2TB Tbolt2 RAID and a 4TB USB external for backup. I generally have the RAID around half full, but if I get buried in a lot of projects-that-refuse-to-die, I have some headroom. Main thing for me is to try to archive in 500MB or so amounts, which is more time-manageable. When the finder says "Estimated time: 9 hours", that's worrisome - our 'hood was built in the 20's and if there's a thundercloud ten miles away, the power goes out (they've improved that now). Backup batteries only last so long and I have a lot of stuff plugged into the UPS.

Again, these sorts of workflows are insanely unique to the freelancer or mega-huge effects shop or first-year DSLR shooter, and every case is probably different. A HUGE driver for me is cost and ROI - I'n in an enviable position of mostly working on fun stuff for good, long-term clients and I don't have to go out and beat the bushes every week - but that means I don't work some days at all. It's interesting to me to read the different solutions people use.