Building a (complex) 4K Video Editing Machine that's also Quiet

[jcs]

Our 2010 2.93GHz 12-core MacPro with GTX 980ti couldn't edit complex 4K footage smoothly in Premiere Pro CC (in OSX El Capitan or Win10x64Pro). FCP X for the most part can handle medium complexity 4K (chroma key, multiple 4K streams at once, very high resolution stills (Ken Burns etc.). However, I'm quite a bit faster in PP CC, and I need to research plugins to replicate what I can do with Lumetri (if such a plugin exists). It was time to upgrade the video editing machine, and it seemed like building a Hackintosh would create a far faster machine than anything Apple currently has to offer.

The goal was to balance high performance with low noise on air cooling only (tricky!). Here's the machine:

• Corsair Carbide Series 330R Blackout Edition Ultra-Silent Mid-Tower Case
• ASUS X99-DELUXE II LGA 2011-v3 ATX Motherboard (ASUS has fancier boards however this appears to be just as fast)
• Intel Core i7-6950X 3.0 GHz Ten-Core LGA 2011-v3 Extreme Edition Processor (have been buying or building dual CPU Xeons for over 10 years. The single chip high clock 10 core looked to be the best fit for video editing vs. lower clock more core Xeons)
• Corsair Vengeance LPX 64GB (4x16GB) DDR4 3466 (PC4-27700) C16 with Airflow, Black CMK64GX4M4B3466C16
• Noctua Dual Tower CPU Cooler for Intel LGA 2011-0/LGA 2011-3 Square ILM/1156/1155/1150 and AMD AM2/AM2+/AM3/3+,FM1/2 NH-D9L
• Noctua AAO Frame Design, SSO2 Bearing Premium Quality Quite Fan NF-A9 PWM (Extra fan for CPU cooler)
• ASUS GeForce GTX 1080 8GB ROG STRIX Graphics Card (STRIX-GTX1080-A8G-GAMING) (purchased over eVGA (normally favorite) due to heat issues with ACX 3 and the VRM chip)
• Corsair ML120 Pro, 120mm Premium Magnetic Levitation Cooling Fan CO-9050040-WW (to replace rear case fan)
• Corsair ML140, 140mm Premium Magnetic Levitation Fan (2-Pack) (to replace front fan and add additional fan to cool hard drives)
• Samsung 2TB 960 PRO M.2 Internal SSD (fastest drive available (~3.5GB/s), for OS, Apps, temp projects)
• WD 4TB Black 7200 rpm SATA III 3.5" Internal HDD (long term storage for video projects)
• EVGA SuperNOVA 1000 P2 1000W Power Supply (higher quality PSU's increase component life and reliability)
• Microsoft Windows 10 Pro (64-bit, OEM DVD) (lower cost than download!)
• SanDisk 16GB Extreme USB 3.0 Flash Drive (downloaded Win10 from Microsoft and used key from DVD to create USB installer)

I was able to get it stable at 3.96GHz (in turbo/boost) with core temps max around 65-69C (all fans running 100%, fan speed temp ramping tuned in bios). Memory was only stable at 2666 (so much for 3466 rating). It will run at 4-4.1GHz, however some cores temp spike to 80C, so is probably too much for this air cooling set up.

The latest NVidia drivers are still buggy with Premiere- the "black screen bug with Ultra key" is still present. I installed the latest CUDA development video driver figuring it might be more stable than the game-oriented drivers (black screen bug hasn't appeared in latest green screen project yet). 4K editing in software mode is pretty smooth (full res preview)! So if the GPU driver has severe bugs during a project I can switch to software mode and still work reasonably fast.

Windows 10's font rendering looks much crisper than El Capitan. Window rendering and overall GUI speed is also much faster- feels like you are working faster. Once Windows 10 checked out with Premiere Pro I dug a bit deeper into what it would take to install OSX (would test on a hard drive then replace with SSD). Here's the closest I found for this motherboard (uses apparently the same bios). Doable however a lot of time hunting down kext (kernel extensions) and trial and error. So for the time being the new machine is Windows only- I can access both machines through a USB switch for keyboard/mouse (monitors have dual inputs).

At idle this machine is nearly silent, much quieter than the 2010 MacPro. It's good to see that after 6 years there's been some progress in CPUs and memory (GPUs have had much more performance growth). The new NVMe PCI SSD drives are insanely fast, however 4K editing straight from a USB3 port and CFast 2.0 works fine (as does editing from a hard drive over SATA- not the bottleneck).

Hopefully this info will be helpful for anyone looking to build a fast and quiet 4K editing box.

 

[pablogrollan]

Looking at your build the only thing that stands out is the lack os separate drives. You'll need more than two if you want to edit 4K smoothly... I'd recommend 4. As for the "quiet" feature, going all SSD helps. I've only had SSDs for 3 years now and I'm very happy with the result: silent, fast and reliable. In fact, having four separate SSDs for editing you don't need them to be PCI-E, SATA would do -they are faster than required to edit 4K-.

I still cannot understand how come you old build could not handle 4K properly. I don't know if you had a configuration issue, an undetected bottleneck or if simply Premiere's performance is way worse on Mac, but 12 cores and a GTX 980ti should usually be enough.

[tugela]

It isn't really 12 cores, it is two processors, each with 6 cores. They may not be being fully/efficiently utilized. His modern build has a current single processor with 10 cores. The microcode likely runs more efficiently in that than the 2010 processors.

[joema]

4 hours ago, jcs said:

Our 2010 2.93GHz 12-core MacPro with GTX 980ti couldn't edit complex 4K footage smoothly in Premiere Pro CC....FCP X for the most part can handle medium complexity 4K (chroma key, multiple 4K streams at once, very high resolution stills (Ken Burns etc.)....it seemed like building a Hackintosh would create a far faster machine than anything Apple currently has to offer...The goal was to balance high performance with low noise on air cooling only (tricky!). Here's the machine...

Thanks for providing all that information. Max Yuryev recently did something similar on a lower-end Hackintosh optimized for FCPX video editing, although you could also run Premiere on it. He built a $1k quiet machine that is faster than a 6-core D700 2013 Mac Pro. It used a heavily overclocked 4-core CPU and a sealed water cooling kit. His goal was minimum software driver tweaking so he used dual AMD 280X GPUs -- essentially identical to the D700 -- rather than more recent nVidia cards. FCPX is also optimized for those cards. For Premiere you'd obviously replace those with higher-end nVidia GPUs. Overall not really at the performance level of your build, but it was quiet, inexpensive, and plenty fast for 4k editing on FCPX. He provides many benchmarks in this video.

It would be interesting if you could somehow run some of the same benchmarks he ran and compare them.

 

[Taranis]

I've been told that for long term storage it's generally better to choose low rpm drives, they are supposed to be more reliable. Most of the NAS drives are low rpm. I never cared but I thought it's worth mentioning

[afs]

nice config!!

I would only consider noctua d15 instead of d9 (better temps and even quieter).

Super flower leadex platinium/titanium is also good choice but both are excellent, top tier power supply.

[no_connection]

I would have liked ECC memory, but today's memory is getting good enough I guess/hope.
The main reason for not using two or more CPUs is they don't share RAM. So it have to copy/move data between RAM banks if it wants to use a thread on another CPU. That may or may not be a problem depending on workload.

As for "long term storage" I would use a filesystem with checksum and correction. Mainly ZFS running on FreeNAS. Data corruption is bad to have, not knowing about it even worse.
Especially if .mov head or tail get corrupted ruining the entire clip.

[Huuow]

Insane machine you bought there ^^

But before anyone throws something out of the window by frustration... a computer for instance... Be sure that performance problems were not infact software related.. i ran into this piece of crap on my current project: https://forums.adobe.com/thread/2193382

Quite common it seems and has especially problems with 4k and lumetri filters.... had to upgrade to cc 2017 as only fixable solution..

[benymypony]

And what about this (extreme) solution?  

 

[jcs]

21 hours ago, pablogrollan said:

Looking at your build the only thing that stands out is the lack os separate drives. You'll need more than two if you want to edit 4K smoothly... I'd recommend 4. As for the "quiet" feature, going all SSD helps. I've only had SSDs for 3 years now and I'm very happy with the result: silent, fast and reliable. In fact, having four separate SSDs for editing you don't need them to be PCI-E, SATA would do -they are faster than required to edit 4K-.

I still cannot understand how come you old build could not handle 4K properly. I don't know if you had a configuration issue, an undetected bottleneck or if simply Premiere's performance is way worse on Mac, but 12 cores and a GTX 980ti should usually be enough.

The NVMe PCIe drive is 3.5GB/s read- mostly for app loading speed. I tend to edit off of 2TB WD USB3 external drives and rotate them out when full- they are plenty fast for compressed 4K files (FCPX has not problem, PP CC is slow/choppy unless using proxies and/or 1/2 or 1/4 playback res). I also have a 4 drive 3.5 HD cartridge system for backups (eSATA and USB3).

You're right, a 12-Core 2.93GHz and GTX 980ti should be more than fast enough, and it is with FCPX (easily). Watching RAM usage on the new machine playing 4K material I noticed that PP CC used up 20GB of RAM. The 12-Core has only 24GB of RAM. Might try upping the RAM to 64GB ($319 from macsales.com). Would be useful to have 2 machines 4K capable (back up in case of hardware issue with one machine etc.). Also, sometimes there are bugs in PP CC in Windows and not OSX and vice versa, useful to be able to run either OS.

18 hours ago, joema said:

Thanks for providing all that information. Max Yuryev recently did something similar on a lower-end Hackintosh optimized for FCPX video editing, although you could also run Premiere on it. He built a $1k quiet machine that is faster than a 6-core D700 2013 Mac Pro. It used a heavily overclocked 4-core CPU and a sealed water cooling kit. His goal was minimum software driver tweaking so he used dual AMD 280X GPUs -- essentially identical to the D700 -- rather than more recent nVidia cards. FCPX is also optimized for those cards. For Premiere you'd obviously replace those with higher-end nVidia GPUs. Overall not really at the performance level of your build, but it was quiet, inexpensive, and plenty fast for 4k editing on FCPX. He provides many benchmarks in this video.

It would be interesting if you could somehow run some of the same benchmarks he ran and compare them.

 

That's a cool build using parts known to work with Uni/Multibeast/Clover etc. Super easy install for that hardware. X99 is newer and takes a lot more work to find all the right pieces and edit all the right files. Probably not that hard once all the pieces/configs found though  For real-time editing the dual AMD 280X's and GTX 1080 might be pretty close, for rendering the 10 core will be a bit faster

11 hours ago, no_connection said:

I would have liked ECC memory, but today's memory is getting good enough I guess/hope.
The main reason for not using two or more CPUs is they don't share RAM. So it have to copy/move data between RAM banks if it wants to use a thread on another CPU. That may or may not be a problem depending on workload.

ECC is only supported on Xeon processors and it's also a bit slower. Unless one is in space (cosmic rays), non-ECC is pretty reliable (PP CC bugs far outweigh possible memory issues ;)). Each processor in a dual CPU system can access all the memory using e.g. QPI with little overhead: https://www.quora.com/On-a-dual-CPU-system-Xeon-can-a-single-processor-access-all-the-memory . Would be interesting to see how a modern dual 6 core Xeon (using 10 cores) at the same clock speed as I7 10 core compares (perhaps down clocked).

 

8 hours ago, Huuow said:

Insane machine you bought there ^^

But before anyone throws something out of the window by frustration... a computer for instance... Be sure that performance problems were not infact software related.. i ran into this piece of crap on my current project: https://forums.adobe.com/thread/2193382

Quite common it seems and has especially problems with 4k and lumetri filters.... had to upgrade to cc 2017 as only fixable solution..

Yeah we ran into that and installed the prior version to work around the issue (happened in both Windows and OSX). Using 2017 on new machine, so far so good (tests only, will start editing real projects tomorrow).

 

4 hours ago, benymypony said:

And what about this (extreme) solution?  

Haha pretty cool! Doesn't really run faster; too bad fish can't live in mineral oil  Liquid nitrogen or helium is pretty neat (5.7 (6950x) to 8+GHz (5 years ago) overclocking).

14 hours ago, afs said:

nice config!!

I would only consider noctua d15 instead of d9 (better temps and even quieter).

Super flower leadex platinium/titanium is also good choice but both are excellent, top tier power supply.

Thanks! I read the D15 wouldn't fit in the 330R case:http://www.tomshardware.com/answers/id-2966620/noctua-d15s-fit-corsair-330r-case.html . It looks like the eVGA SuperNOVA P2 uses the Super Flower Leadex Platinum platform http://www.tomshardware.com/answers/id-2523573/super-flower-psu.html?

[OliKMIA]

Very good machine but a few remarks just for the sake of it:

• Corsair Carbide Series 330R Blackout Edition Ultra-Silent Mid-Tower Case. >>> The fractal Design R5 is also very good and easy to setup
• Corsair Vengeance LPX 64GB (4x16GB) DDR4 3466 (PC4-27700) C16 with Airflow, Black CMK64GX4M4B3466C1. >>> Enormous Heat-sink on RAM are useless except for marketing purpose. High frequency also has very little impact in real life even if the gains are visible in the benchmarks. A basic Kingston or Crucial DDR4 2400 will do the job.
• Samsung 2TB 960 PRO M.2 Internal SSD (fastest drive available (~3.5GB/s), for OS, Apps, temp projects). >>> Given this beast computer I would get 2 SSD, one for the system (windows) and one for the hot files currently in the timeline
• WD 4TB Black 7200 rpm SATA III 3.5" Internal HDD (long term storage for video projects).  >>> I got 2 similar HD but in RAID 1 since they are my dump area waiting for another round of saving.
• EVGA SuperNOVA 1000 P2 1000W Power Supply (higher quality PSU's increase component life and reliability). EVGA is good brand but 1000W is a complete overkill. 650W would have been is plenty already for this machine unless you plan on doing SLI later on.

PS: yeah, the new CC 2017 is buggy during rendering with Lumetri and many export tend to crash. If that happens I found that exporting with AME and selecting the Mercury/Software/CPU option instead of the GPU solves the issue (might take longer to render though)

[pablogrollan]

2 hours ago, jcs said:

I tend to edit off of 2TB WD USB3 external drives and rotate them out when full- they are plenty fast for compressed 4K files

Yet I'd say that's a clear bottleneck there... Never edit off external USB3 drives, and even less if they are mechanical drives. That's an "on the field" solution, but you'll find editing with internal SSDs is much better (faster, more responsive). Also using only 2 drives is another bottleneck.

It's great to have the software running on the NVMe PCIe drive, but kind of useless; I guess the OS and Premiere load crazy fast, but once Premiere is loaded, it's loaded -and the drive becomes irrelevant-. It's crucial to have your software/OS on one drive, your source footage on another and your render/temp files on another so that you can use all of the drives at the same time and maximize the SATA bus. Keep in mind spinning discs have a slower access time (regardless of their read speed), especially when they have to access two pieces of data in faraway sectors at the "same" time. It wouldn't be so obvious with a single SSD editing drive because access time is much faster, but you'd still be able to tell the difference between a system with a single editing SSD and one with several HDDs accessed at the same time. It's better/faster to have 4 or 5 "slow" HDDs than a single SSD.

2 hours ago, jcs said:

You're right, a 12-Core 2.93GHz and GTX 980ti should be more than fast enough, and it is with FCPX (easily). Watching RAM usage on the new machine playing 4K material I noticed that PP CC used up 20GB of RAM.

That's a Mac problem. PP uses a lot of RAM (the whole Adobe suite does) and in Windows it is much easier to control what is open and running in the background. MacOS is notorious for having tons of processes and apps running in the background. Since macs are usually overkill, the user experience is great -we've all heard people say that apps open much faster unaware that they were never really actually closed-, but when you want to push it to the limit it just doesn't deliver like a windows PC does. It's just a question of knowing the limitation and accounting for the necessary overhead. I recently upgraded the GPU in an old editing suite (GTX 1070, i7, 32GB RAM) and it edits 4K without proxies and without hiccups. It's not as fast is some processes as the new suites, but does the job without crashing/lagging or any of that.

You'll see that a single fast processor is better for Premiere. It doesn't really take advantage of dual Xeons like a 3D app does. Still, with such a beast you should consider a multiple drive configuration. You can go the HDD route (RAID) which would require many disks -but cheaper- or go all SSD which would require just 4 SSDs since there is no need for RAID configuration.

[Kisaha]

I have a Xeon 1230/3400MHz, 16GB RAM, a SSD and a series of Western Black, and I GTX 660 (2GB).

Does anyone think that with a 1060/3GB can edit 4K sufficiently (with 1/2 preview), or do I have to build a new one?

[Don Kotlos]

4 hours ago, Kisaha said:

I have a Xeon 1230/3400MHz, 16GB RAM, a SSD and a series of Western Black, and I GTX 660 (2GB).

Does anyone think that with a 1060/3GB can edit 4K sufficiently (with 1/2 preview), or do I have to build a new one?

It depends on the 4K files & the effects that you use. If you use proxies you might be fine with just a GPU upgrade. 

See here for a series of tests with premiere:

 

[jcs]

20 hours ago, pablogrollan said:

Yet I'd say that's a clear bottleneck there... Never edit off external USB3 drives, and even less if they are mechanical drives. That's an "on the field" solution, but you'll find editing with internal SSDs is much better (faster, more responsive). Also using only 2 drives is another bottleneck.

It's great to have the software running on the NVMe PCIe drive, but kind of useless; I guess the OS and Premiere load crazy fast, but once Premiere is loaded, it's loaded -and the drive becomes irrelevant-. It's crucial to have your software/OS on one drive, your source footage on another and your render/temp files on another so that you can use all of the drives at the same time and maximize the SATA bus. Keep in mind spinning discs have a slower access time (regardless of their read speed), especially when they have to access two pieces of data in faraway sectors at the "same" time. It wouldn't be so obvious with a single SSD editing drive because access time is much faster, but you'd still be able to tell the difference between a system with a single editing SSD and one with several HDDs accessed at the same time. It's better/faster to have 4 or 5 "slow" HDDs than a single SSD.

Normally I'd agree regarding external USB drives, however USB 3 is really fast, and the WD Ultra 2TB drives are rated at 110MB/s read (and get about that for real: https://www.youtube.com/watch?v=z8p03IjCclA ). 4K C300 II files are around 440Mbps or ~55MB/s which gives us around 2x overhead which matches my experience as well. 1DX II 4K60p is ~800Mbps or 100MB/s and those work as well too. Transitions might cause a burst slowdown, however since reading is faster than the file bitrate the system can catch up in most cases (my real world experience). For something more complex, sure, an SSD or HD RAID 0 is needed. Our 4K complexity comes from green screen, compositing, visual effects, large images, etc. (large images can also choke/expose PP CC bugs). I would expect the 3.5GB/s (3.5 GIGABYTE per sec!) NVMe PCIe drive to blow away just about any multidrive solution except perhaps 2 NVMe drives in RAID 0 ;). There is no seek overhead, it's RAM and PCIe is faster than SATA (latency), and it's going to be faster than multiple spinning disks with seek latency (when total bandwidth is the same or more). Loading apps in a blink is pretty cool (and does save time- it adds up), and with 2T there's enough room for editing a full project 100% on the drive (our episodes are less than 30 minutes finished). Once released the project can be moved to a 3.5 HD archive (4 drive dockers are handy- can fill drives and put in a case then take offline and store). 4 Samsung 850 Pro would be ~2.2GB/s  RAID 0, still getting smoked by the single 960 Pro at 3.5GB/s (except in total storage).

 

20 hours ago, Kisaha said:

I have a Xeon 1230/3400MHz, 16GB RAM, a SSD and a series of Western Black, and I GTX 660 (2GB).

Does anyone think that with a 1060/3GB can edit 4K sufficiently (with 1/2 preview), or do I have to build a new one?

It depends on the 4K material: GH4 tends to be easier, A7S2 harder, and C300 II and 1DX II 60p the hardest. Perhaps also consider trying more RAM (e.g. 64GB) before getting a new machine. WD Black is plenty fast unless you're compositing a bunch of 4K videos in the same frame (transitions will be fine).

 

[estarkey7]

11 minutes ago, jcs said:

Normally I'd agree regarding external USB drives, however USB 3 is really fast, and the WD Ultra 2TB drives are rated at 110MB/s read (and get about that for real: https://www.youtube.com/watch?v=z8p03IjCclA ). 4K C300 II files are around 440Mbps or ~55MB/s which gives us around 2x overhead which matches my experience as well. 1DX II 4K60p is ~800Mbps or 100MB/s and those work as well too. Transitions might cause a burst slowdown, however since reading is faster than the file bitrate the system can catch up in most cases (my real world experience). For something more complex, sure, an SSD or HD RAID 0 is needed. Our 4K complexity comes from green screen, compositing, visual effects, large images, etc. (large images can also choke/expose PP CC bugs). I would expect the 3.5GB/s (3.5 GIGABYTE per sec!) NVMe PCIe drive to blow away just about any multidrive solution except perhaps 2 NVMe drives in RAID 0 ;). There is no seek overhead, it's RAM and PCIe is faster than SATA (latency), and it's going to be faster than multiple spinning disks with seek latency (when total bandwidth is the same or more). Loading apps in a blink is pretty cool (and does save time- it adds up), and with 2T there's enough room for editing a full project 100% on the drive (our episodes are less than 30 minutes finished). Once released the project can be moved to a 3.5 HD archive (4 drive dockers are handy- can fill drives and put in a case then take offline and store). 4 Samsung 850 Pro would be ~2.2GB/s  RAID 0, still getting smoked by the single 960 Pro at 3.5GB/s (except in total storage).

 

It depends on the 4K material: GH4 tends to be easier, A7S2 harder, and C300 II and 1DX II 60p the hardest. Perhaps also consider trying more RAM (e.g. 64GB) before getting a new machine. WD Black is plenty fast unless you're compositing a bunch of 4K videos in the same frame (transitions will be fine).

 

The speed is not relevant here, it's the quantity of request to the drive. USB is not truly designed for this type of access pattern. There are many layers of drivers on top of USB before you get to the Mass Storage profile, and that reduces performance. You'd benefit a whole bunch with internal hard drive.

[jcs]

2 hours ago, estarkey7 said:

You'd benefit a whole bunch with internal hard drive.

How about a Samsung 960 Pro?

[pablogrollan]

57 minutes ago, jcs said:

How about a Samsung 960 Pro?

Nope! I know it seems you have plenty of overhead with those 3.5 GB/s, but regardless of what specs say I'm talking from the experience of editing simple FullHD on a top Macbook Pro with the SM951 SSD (2 GB/s) and having OS, source media, temp files, renders and exports on the same drive always results in a less responsive editing experience, not to mention the risk... Should anything happen to that drive you would lose the entire editing suite. It could be an acceptable compromise exceptionally and out on the field, but IMHO not on a desktop station, though I suppose it depends on what you need it for -I make my livelihood out of it, so minimizing the risks is essential-.

[jcs]

On 1/4/2017 at 2:32 AM, pablogrollan said:

Nope! I know it seems you have plenty of overhead with those 3.5 GB/s, but regardless of what specs say I'm talking from the experience of editing simple FullHD on a top Macbook Pro with the SM951 SSD (2 GB/s) and having OS, source media, temp files, renders and exports on the same drive always results in a less responsive editing experience, not to mention the risk... Should anything happen to that drive you would lose the entire editing suite. It could be an acceptable compromise exceptionally and out on the field, but IMHO not on a desktop station, though I suppose it depends on what you need it for -I make my livelihood out of it, so minimizing the risks is essential-.

Minimizing risk makes total sense. In terms of performance, I was able to edit 4K C300 II files (around 410Mbps) on a 7200 RPM HD over 1Gbps network smoothly.

In Windows PP CC 2017 seems a little faster with 4K (GTX 980ti, 12 Core 2.93).

[estarkey7]

On 1/4/2017 at 4:24 AM, jcs said:

How about a Samsung 960 Pro?

It's one of the best. Even better if you can get it in M2 format.