A story about 4K XAVC-S, Premiere and transcoding

[dafreaking]

So what seems to be the final consensus on this? Convert 4K 100M files to ProRes LT  or HQ or Regular? 

[Matt Holder]

My Imac FCPX  edits GH4 4k like its 1080p - natively and like butter.  

I put some A7s II 4k footage in there and it was stall city - no creative flow on the timeline - i hated it! Seriously bad implications for workflow, productivity and client turn around.

Tip of the hat to Panasonic for an awesome easy to edit 4k codec.

i am sure transcoding would sort the A7s footage out but its a pain...

 

[joema]

3 hours ago, dafreaking said:

So what seems to be the final consensus on this? Convert 4K 100M files to ProRes LT  or HQ or Regular? 

There is no simple answer. Some systems can edit the camera native files with good performance for one stream. Most systems cannot do this smoothly for 4k H264 multicam, and some type of transcoding is needed, whether externally before ingest or to proxy during/after ingest. Fortunately Premiere now supports this and gives various resolution and codec options for proxy, including H264, Cineform, and ProRes 422. FCPX always transcodes to 1/4 res ProRes 422, e.g, 1080 from 4k.

Also (as already mentioned) not all 4k H264 codecs are the same. Some may exhibit smoother editing on certain software.

For documentary projects with a large shooting ratio, it is nice (in FCPX) to skim through the camera native files without transcoding all that. For scripted narratives or other content with a lower shooting ratio, the workflow might favor transcoding everything up front or possibly doing initial selects outside the editor before import.

Some groups mandate ProRes recording off the camera, so all their cameras either do this internally or have external recorders. Others do the initial evaluation and selection using camera native files. Still others transcode to a mezzanine codec before ingest. It depends on the equipment, preferences and workflow policies of the group. My group can shoot a terabyte of 4k H264 per weekend so we don't transcode to ProRes before or after ingest since that would be at least 8 terabytes. We selectively transcode to proxy after ingest if needed for 4k multicam.

[Phil A]

I get occassional lag running the native files from a Sony a6300 in Davinci Resolve on my system, too.

i7 5930k @ 4.5 Ghz, 32GB Ram, AMD R9 390 and all files on a bunch of Samsung 850 Pro SSDs

 

I think the best way is to either use the proxy function of your editor or first sort out the shots you want to use and then transcode them to ProRes or DNxHD.

[dafreaking]

6 hours ago, Phil A said:

I get occassional lag running the native files from a Sony a6300 in Davinci Resolve on my system, too.

i7 5930k @ 4.5 Ghz, 32GB Ram, AMD R9 390 and all files on a bunch of Samsung 850 Pro SSDs

 

I think the best way is to either use the proxy function of your editor or first sort out the shots you want to use and then transcode them to ProRes or DNxHD.

I'm going to do some tests today if I get the chance on my slow ass laptop to transcode some footage. Since the motheboard of my main rig is in the shop this is going to be painful. I guess I need a new laptop super soon...grrr.

 

People with real world experiences. How much of a difference (if any) has putting media files on SSDs made for you? More specifically timeline performance, i.e. scrubbing through footage etc.

[joema]

6 hours ago, dafreaking said:

...People with real world experiences. How much of a difference (if any) has putting media files on SSDs made for you? More specifically timeline performance, i.e. scrubbing through footage etc.

I have six Macs, three with Fusion Drive and three with SSD. While my media content is usually on external Thunderbolt arrays, I have done lots of testing with smaller projects on SSD. I don't see much performance difference attributable to I/O if editing H264. In hindsight this should be obvious -- if the I/O rate was that high, the puny CPU and I/O system in the camera could not write it to storage fast enough. Anyone who doubts this can simply monitor I/O rates when editing H264 content by using Activity Monitor or Windows Performance Monitor -- they aren't that high.

SSD can make a difference if editing lower-compression codecs like ProRes. In that case the I/O rate can be 8x or 10x the camera native rate -- for a single stream. For three-camera multicam it could be 30x the camera native rate. In that case you may really need the additional I/O performance, but SSD is often too small or too expensive in those cases. 

[dafreaking]

So my main objective is probably using a SSD as a media drive for onsite edits for 70% XAVC-S 1080p (50M) and 30% XAVC 4K-S (100M). As these are time sensitive (things like same day edits and next day edits) would an SSD be worth it? or should I get something like a 1TB WD Black (Mobile) and stuff it in an enclosure? The editing will be performed on a decently specced Macbook Pro using Premiere.

Edit:

 

Will also use it to edit off an i7-5820K based workstation when in the office.

[joema]

7 hours ago, dafreaking said:

So my main objective is probably using a SSD as a media drive for onsite edits for 70% XAVC-S 1080p (50M) and 30% XAVC 4K-S (100M). As these are time sensitive (things like same day edits and next day edits) would an SSD be worth it? or should I get something like a 1TB WD Black (Mobile) and stuff it in an enclosure? The editing will be performed on a decently specced Macbook Pro using Premiere.

Almost any 7200 rpm 3.5" drive would work for this, but they are externally-powered, hence not very convenient for portable use. For 1080p, it's no problem from a CPU or I/O standpoint. I edit a lot of 4k XAVC-S, and for camera native the data rate isn't that high. However the CPU load is very high, especially for Premiere. This leads to transcoding to proxy (a CPU-bound operation) which takes time and increases I/O load when completed, since the video files are much less dense.

If you want portability, then staying with a bus-powered drive is nice but most USB 3 bus-powered drives are too slow, IMO. The 4TB Seagate Backup Plus Fast is bus-powered, only about $185, and it's pretty fast (internally RAID-0): https://amzn.com/B00HXAV0X6 I have several of those and they work well. Below are other bus-powered external SSD options I don't have personal experience with.

Lacie 1TB Thunderbolt bus-powered SSD ($900): https://eshop.macsales.com/item/Lacie/9000602/

Transcend 1TB Thunderbolt bus-powered SSD ($589): https://amzn.com/B00NV9LTFW

If USB 3 is OK, this 1TB bus-powered external SSD is about $400: https://eshop.macsales.com/item/OWC/ME6UM6PGT1.0/

[Asmundma]

On 11/6/2015 at 11:23 AM, Andrew Reid said:

Thanks Don nice post.

Meanwhile on a Mac I recommend to simply use EditReady, transcode to ProRes 422 LT and you will be able to playback full quality full res 4K on a laptop. There's no quality loss from XAVC-S.

The only way you can lose quality from XAVC-S is to go backwards significantly from H.264 100Mbit/s and that is quite hard to do

Why not use FCPX, there no problem editing full resolution 4K  xavc-s files. It also faster to use for editing as well (after some learning). Renders faster according to many tests.

if your on Mac, it's a no brainer. 

On 7/18/2016 at 11:03 AM, Matt Holder said:

My Imac FCPX  edits GH4 4k like its 1080p - natively and like butter.  

I put some A7s II 4k footage in there and it was stall city - no creative flow on the timeline - i hated it! Seriously bad implications for workflow, productivity and client turn around.

Tip of the hat to Panasonic for an awesome easy to edit 4k codec.

i am sure transcoding would sort the A7s footage out but its a pain...

 

No problem here with 4K A7s2 files....and FCPX 

[joema]

14 hours ago, Asmundma said:

Why not use FCPX, there no problem editing full resolution 4K  xavc-s files. It also faster to use for editing as well (after some learning). Renders faster according to many tests.

if your on Mac, it's a no brainer. 

No problem here with 4K A7s2 files....and FCPX 

As a former experienced Premiere editor who moved to FCPX, this can be a difficult transition. It's not like moving between other track-based editors such as Avid or Vegas. The paradigm is radically different, and for some users entails a lengthy learning curve. They are both good products. It's true FCPX is faster at various things on the same hardware but whether this produces the end product any faster is more complex. Premiere users often depend on After Effects or other components of the Adobe suite, so it's often easier to stay with that. They may be part of a workgroup so changing editing software is not an individual decision.

Re 4K XAVC-S, I have terabytes of this and while FCPX on a top-spec iMac 27 can handle a single stream without transcoding to proxy, it still it still requires proxy for smooth editing of 4K H264 multicam.

Premiere users on Windows can easily build or buy whatever hardware they need to obtain good performance. On Mac the options are more limited. However since Premiere now has integrated proxy support, that will solve most performance problems, at the time and space cost of transcoding the files. However the transcode is a background process so you can continue to work while that runs.

[Asmundma]

18 hours ago, joema said:

As a former experienced Premiere editor who moved to FCPX, this can be a difficult transition. It's not like moving between other track-based editors such as Avid or Vegas. The paradigm is radically different, and for some users entails a lengthy learning curve. They are both good products. It's true FCPX is faster at various things on the same hardware but whether this produces the end product any faster is more complex. Premiere users often depend on After Effects or other components of the Adobe suite, so it's often easier to stay with that. They may be part of a workgroup so changing editing software is not an individual decision.

Re 4K XAVC-S, I have terabytes of this and while FCPX on a top-spec iMac 27 can handle a single stream without transcoding to proxy, it still it still requires proxy for smooth editing of 4K H264 multicam.

Premiere users on Windows can easily build or buy whatever hardware they need to obtain good performance. On Mac the options are more limited. However since Premiere now has integrated proxy support, that will solve most performance problems, at the time and space cost of transcoding the files. However the transcode is a background process so you can continue to work while that runs.

I  have used Vegas and Premiere before - but there is no doubt that FCPX editing is fastest (for me). I am also very experienced in DAW - eg- Reaper and latest years Logic, so in that respect use to track based software.  

I can run 3 4k streams - it start to struggle on the 4th.   top spec iMac, but only 500MB/s SSD, not that one with over 1000MB/s  (very latest model) 

Else agree about the eco system - we're not alone. 

[dafreaking]

On 19/07/2016 at 3:14 AM, joema said:

Almost any 7200 rpm 3.5" drive would work for this, but they are externally-powered, hence not very convenient for portable use. For 1080p, it's no problem from a CPU or I/O standpoint. I edit a lot of 4k XAVC-S, and for camera native the data rate isn't that high. However the CPU load is very high, especially for Premiere. This leads to transcoding to proxy (a CPU-bound operation) which takes time and increases I/O load when completed, since the video files are much less dense.

If you want portability, then staying with a bus-powered drive is nice but most USB 3 bus-powered drives are too slow, IMO. The 4TB Seagate Backup Plus Fast is bus-powered, only about $185, and it's pretty fast (internally RAID-0): https://amzn.com/B00HXAV0X6 I have several of those and they work well. Below are other bus-powered external SSD options I don't have personal experience with.

Lacie 1TB Thunderbolt bus-powered SSD ($900): https://eshop.macsales.com/item/Lacie/9000602/

Transcend 1TB Thunderbolt bus-powered SSD ($589): https://amzn.com/B00NV9LTFW

If USB 3 is OK, this 1TB bus-powered external SSD is about $400: https://eshop.macsales.com/item/OWC/ME6UM6PGT1.0/

 

Thanks once again for all your insights and input. I've been researching the  4TB Seagate Backup Plus Fast as it seems to tick all boxes. Cheaper than buying a 1 TB SSD, 4 times the space. Just really scared of the stories about the reliability of the drives. 4 TB is also overkill for what I'll be using it for. Thinking that i'll buy 2 x 1TB WD Blacks (portable) and stick them in a RAID 0 bus powered case. It's really between that and the SSD at this point

[tellure]

Curious if anyone has managed to get realtime scrubbing and instant updates when jumping to a new position on the timeline in Premiere Pro with 4K XAVC-S files.  I was hoping to hear that Premiere Pro 2015.3 with the Intel video acceleration would be the solution but all the reports I've read of 2015.3 is that the performance is even worse than before.

[The Chris]

On July 19, 2016 at 7:17 AM, joema said:

I have six Macs, three with Fusion Drive and three with SSD. While my media content is usually on external Thunderbolt arrays, I have done lots of testing with smaller projects on SSD. I don't see much performance difference attributable to I/O if editing H264. In hindsight this should be obvious -- if the I/O rate was that high, the puny CPU and I/O system in the camera could not write it to storage fast enough. Anyone who doubts this can simply monitor I/O rates when editing H264 content by using Activity Monitor or Windows Performance Monitor -- they aren't that high.

SSD can make a difference if editing lower-compression codecs like ProRes. In that case the I/O rate can be 8x or 10x the camera native rate -- for a single stream. For three-camera multicam it could be 30x the camera native rate. In that case you may really need the additional I/O performance, but SSD is often too small or too expensive in those cases. 

You using FCPx or Premiere?

[joema]

3 minutes ago, The Chris said:

You using FCPx or Premiere?

I use both FCPX 10.2.3 and Premiere CC 2015.3.

[Don Kotlos]

An update after the latest premiere CC version:

Using the build in proxy setting of premiere makes editing an enjoyable experience once again. 

They claim better h264 decoding as well but my CPU does not support it. If anyone has an update on that it might be helpful for new system builders. 

[maxmizer]

Prior to editing of 4K XAVC-S files, always use Sony Prepare and save the file to ProRes...It is the easiest way to correctly interpret the file stream.

[mojo43]

On 9/29/2016 at 5:27 PM, Don Kotlos said:

An update after the latest premiere CC version:

Using the build in proxy setting of premiere makes editing an enjoyable experience once again. 

They claim better h264 decoding as well but my CPU does not support it. If anyone has an update on that it might be helpful for new system builders. 

Hi Don, thanks so much for the info in this thread. I am currently feeling the same pain that you were having trying to scrub through 4k footage in Premiere. It is really screwing up my creative process and I hate transcoding. I just don't have the time after a shoot. I am really hoping to find a way to make this work with Premiere. By the way I normally edit in Edius and it has absolutely no problems whatsoever with the same files. So my work around for now is to cut my sections in Edius and when I want to place in Premiere I find the file and the section and place. It really really sucks.

Anyhow, enough venting. Hopefully they will put out an update or something soon..

[Don Kotlos]

Here is a very nice article about optimizing storage configurations with Premiere:

https://www.pugetsystems.com/labs/articles/Adobe-Premiere-Pro-CC-2015-4-Storage-Optimization-854/

One comparing CPU performance:

https://www.pugetsystems.com/labs/articles/Adobe-Premiere-Pro-CC-2015-3-CPU-Comparison-849/

And one comparing GPU performance:

https://www.pugetsystems.com/labs/articles/Adobe-Premiere-Pro-CC-2015-3-Pascal-GPU-Performance-840/