In Apple’s attempt to overturn RED’s claims over visually lossless compressed raw video, the US Patent Office has published documents submitted by RED. These explain their side of the story with particular regard to REDCODE.
If RED can continue to prove that the approach to their codec was novel, RED will win and Apple will have to compensate RED or make a deal in order to sell ProRes RAW in our devices and cameras, such as the Nikon Z6.
First I should make clear that Apple did not “sue” RED, they filed a petition to cancel claims in a patent. As a non-legal person, I have a bit of a “filmmakers” understanding when it comes to the legal system! However I do understand the technical side so let’s take a look at that and the claims RED are making for the continued enforcement of their patent…
Key to the Apple challenge is the novelty of RED’s ‘314 patent, and they are doing what is known in the patent legal system as an inter partes review (seeking to invalidate the patent or some claims of it). Investigators will review prior litigation involving the ‘314 patent vs Sony when RED asserted their claims successfully in a 2013 dispute over RAW recording on the CineAlta cameras, such as the F55.
The main claim Apple is making, and the claim RED is defending, revolves around an arguement that the ‘314 patent should be cancelled because it’s an obvious invention, for a “skilled artisan” to combine “prior art” elements (such as a sensor and recording media) to create the features of those claimed by RED in the ‘314 patent.
In particular Apple points to the teachings of Presler and Molgaard.
Presler’s earlier patent describes a portable digital camera and recording system similar to RED’s designed for high definition raw images at film or video frame rates for HD, 2K and 4K cinema quality production.
Molgaard describes image processing techniques similar to those in RED’s ‘314 patent – namely, “lossless, near-lossless, and lossy compression and decompression of digital image data (optimised for data from a bayer sensor – like the CMOS sensor in all of RED’s cameras to date).
RED must prove their approach to the problem was entirely novel and new, in particular with regards to visually lossless compressed RAW (REDCODE) compared to the systems described in the Preseler and Molgaard patents.
Further, Apple argue “It would have been obvious to combine the specification of the SATA standard… to support the disclosed capability of Presler’s system to store compressed processed images data at a rate of at least about 23 frames per second”.
The SATA interface is described in the RED patent, as the link between the image processor and REDMAG recording media.
RED’s claims
These claims are published in the public domain by the US Patent Office here. The documents are dated 2 days ago, 15th August 2019. Several key people at RED make a declaration, including but not limited to:
- Graeme Nattress, the inventor of REDCODE, RED’s colour science and image processing and currently Lead Camera Systems architect at RED.
- Jarrad Land, President of RED.com LLC and founder of the DVXUser and RedUser forums.
- Jim Jannard, Founder and Chairman of the Board at RED.com
What follows is a huge insight into the very earliest stages of RED’s camera development including the first RED ONE prototype nicknamed “Boris”.
Peter Jackson also features in the claims with a description of his early adoption of the Boris and Natasha prototypes used to shoot “Crossing The Line” in March and April 2007. Jannard explains the initial invitation to New Zealand to deliver the cameras to Jackson for the shoot, keen to emphasise the novelty of the technology in use at that early stage – including the durability of the cameras, portability and most importantly of all “visually lossless video quality”.
Apple claim the earliest filing date for the patent leads back to December 28th 2007. Jannard describes public exhibits in 2006, 2007 and 2008 where he and others inspected the output of the RED ONE prototypes, confirming:
“Upon visual inspection… raw compressed video shot by Boris displayed as visually lossless when decompressed and demosaiced.”
“I also recall that the program used to display Boris’s test footage reported that video taken by Boris achieved 4K resolution at 24 frames per second.”
These statements are backed by images.
Graeme Nattress (nicknamed the Problem Solver in the early project stages) makes his declaration in great detail, describing how the RED project began:
“I first met Mr Jim Jannard in December 2005. At that meeting, we discussed his desire to create the first ever digital motion picture camera that could record compressed digital motion video at cinema quality levels, including 4K. I was intrigued by the possibilities, because combining the ease and flexibility of post-production of digital video while maintaining cinema-quality frame rate and resolution would be a game-changer in the world of movie making. But I also knew that digital compression was highly disfavored in the movie camera industry…
“To solve this problem, one key area I researched was the use of an image sensor with a Bayer-pattern filter. This was one of several unconventional avenues we explored at RED. At the time, such sensors were associated with lower-quality consumer-grade video cameras and derided as incapable of providing cinema-quality video due to artefact and resolution issues. In contrast, the industry consensus held that cinema quality cameras would need to utilize three sensors, with a prism to split red, green and blue light to each sensor. However, we believed that the benefits of a Bayer-pattern image sensor could be optimized if the image data remained in raw, mosaiced format for compression. We believed that such a data workflow could allow the raw digital files to operate as a digital negative, and world provide all of the post-production flexibility of being able to manipulate the original raw data.”
“Immediately following my December 2005 meeting with Mr. Jannard, I began working on the design of RED’s first commercial digital motion picture camera that would become known as the RED ONE. This work would last all throughout 2006 and into 2007 when we commercially launched the RED ONE video camera. My title on the RED ONE project was Problem Solver, and it remains my title to this day.”
“Paragraphs 22-28 of my 2014 declaration explain how we rejected…conventional thinking by implementing known compression techniques, such as JPEG 2000, in such a way as to operate on non-demosaiced Bayer-pattern image data, a type of image data these techniques were not designed to work on.”
“Accordingly, my work on the RED ONE camera was based on a video image processing pipeline that did not include a conventional demosaicing step prior to compression.”
“… The raw Bayer-pattern image data was then sent to a Xilinx processing FPGA chip for pixel correction and processing of the raw Bayer-pattern image data.”
“… The pixel- corrected and processed raw Bayer-pattern video image data was then sent to Analog Devices compression chips, which utilized a mathematically lossy wavelet compression codec known as JPEG 2000, to compress the processed raw Bayer- pattern video image data.”
“… The compressed raw Bayer-pattern image video data was sent to a memory device, by way of a SATA port, for storage as a raw data file that at one point in development was given a .JIM file extension in homage to Mr. Jannard and the new file type that RED had created.”
“We referred to the above programing for the RED ONE cameras, and the resulting raw compressed data files that it generated, as REDCODE.”
“Much of my research and design work that went into the RED ONE focused on the second step above, i.e., how to process the raw Bayer-pattern video image data before compression in way would optimize the decompressed and demosaiced output. This was a key step in REDCODE because, without pre-processing, JPEG 2000 compression was otherwise incapable of processing raw Bayer-pattern image data in a useful way.”
“To solve this problem, I devised and tested a number of methods for processing the raw Bayer-pattern video image data prior to compression. I ultimately devised two methods in particular that were employed on the RED ONE cameras, referred to as GAS (green average subtraction) and pre-emphasis…Â The pre-emphasis processing technique can be used to help preserve image detail. This technique helps tune the compression of pixel values through a pre-emphasis curve…”
“… For example, this technique can be used to spread-apart (increase gradient) pixel values in some ranges of values and squeeze-together (reduce gradient) values in one or more other ranges. The spreading-apart of values preferentially preserves the variations in values, and thus image details, in the ranges where the values are spread apart so that the corresponding details better survive the compression process.”
“In one implementation, the pre-emphasis curve is set to spread apart pixel values corresponding to the darkest image regions, whereas the relatively brightest image regions were squeezed together. This allowed for compression that better preserved the details in the darkest areas of the image, which can otherwise often be washed out.”
“After sufficient testing to find the most ideal curve, keeping in mind the unfixed rendering intent of the raw image data, I settled on a simple power law curve with adjustable black offset. This curve resulted in low complexity and a single tuning variable, among other benefits.”
Summary
In my opinion this is an extremely novel approach from Graeme Nattress. Does Apple have an answer?
USPTO examiners will cast their expert eye over all of the claims and make a decision. That’s a black box for me, and I don’t know what the timeline is or even whether the decision can be appealed. It may be a while before we see the ProRes RAW firmware update on the Nikon Z6.
If Apple fail with their petition, one assumes they will have to make a financial settlement with RED before they can continue to license ProRes RAW. One could also speculate that Apple might want to purchase the company outright if Jim Jannard and the board wish to sell. A deal might also be made around Hydrogen, which as a smartphone may need to license some of Apple’s patents (although that’s just my speculation and I haven’t done research into this yet).
If indeed Apple fail to overturn the ‘314 patent or key aspects of it relating to visually lossless compressed raw, like Sony, they will only have themselves to blame – where were these mighty corporations in 2006?
Much as Jim Jannard’s lawsuit with me in 2010 leaves a bitter taste, I do admire what was achieved in the development of these cameras and especially the work of Graeme Nattress.
Finally as an interesting aside, RED’s defence claims Steven Soderbergh’s “formidable professional reputation” is staked upon the visual quality of his films…
… Pointing out that “Artifacts from digital compression are simply unacceptable.”
They did not mention Steven Soderbergh’s 2018 film Unsane… Which was shot on an iPhone 🙂
