Camera resolution myths debunked

[HockeyFan12]

7 hours ago, jonpais said:

I was under the impression that the clients of most members here weren't demanding 4K anyhow. Or am I mistaken?

As of last year that was pretty much true for me. This year, I'm being asked to work on an increasing amount of 4k content. 

[jcs]

9 hours ago, meanwhile said:

I meant that there was as much truth in the claims that this tech can replace raw as there is juice in a balloon. (What with balloon's being notably juiceless if operated correctly..)

This is true in the sense that we can't do very much, yes. However by your phrasing you've tried to imply inevitable progress. This is not how competent people reason or argue: if they believe there is a definite reason you should expect progress in a field, they state it.

In this case, you are expecting something especially ridiculous - that ML can come up with a compact description thousands of times more efficient than those currently used for generative computer graphics. Which, honestly, is silly.

Ok: I've told you I've actually done research in this field and you expect me to have my mind changed by the most childish possible example. Does this make sense? (Hint: the answer is NOT "yes"...)

...Arguing that because we can generate pretty fractals we can use ML to replace raw is like saying "I can grow pretty blue crystals in copper sulphate solution! Therefore if collect the juice from the Sunday roast for a few weeks I can clone a human being in it!!!" One thing has nothing to with the other. Thinking that they do requires a level of intellectual resolution so low that - well, so low that I actually can't come up with a useful phrase you'd be likely to understand.

...You've also responded to an argument that ML systems can't reason surpass human generative graphics by showing examples of human generative graphics. This is also silly. And it's even sillier because another point I made was that human created generative graphics don't do hair or skin well and you've shown examples of robots(?)

It's fine to be excited by things, but speaking as someone who actually knows this field, then - for the benefit of everyone else, not you - this ain't gonna happen! 

Have you noticed how @HockeyFan12 has disagreed with me politely in this thread, and we've gone back in forth in a friendly manner as we work through differences in ideas and perceptions, for the benefit of the community?

The reason you reverted to ad hominem is because you don't have a background in mathematics, computer graphics, simulations, artificial intelligence, biology, genetics, and machine learning? That's where I'm coming from with these predictions: https://www.linkedin.com/in/jcschultz/. What's your linkedin or do you have a bio page somewhere so I can better understand your point of view? I'm not sure how these concepts can be described concisely from a point of view solely from chemistry, which appears to be where you are coming from? Do you have a link to the results of your research you mentioned? It's OK if you don't have a background in these fields, I'll do my best to explain these concepts in a general way.

I used the simplest equation I am familiar with, Z^2 + C, to illustrate how powerful generative mathematics can create incredibly complex, organic looking structures. That's because nature is based on similar principles. There's an even simpler principle, based on recursive ratios: the Golden Ratio: https://en.wikipedia.org/wiki/Golden_ratio. Using this concept we can create beautiful and elegant shapes and patterns, and these patterns are used all over the place, from architecture and aesthetic design to nature in all living systems:

 

I did leave the door open for a valid counter argument, which you didn't utilize, so I'll play this counter argument which might ultimately help bridge skepticism that generative systems will someday (soon) provide massive gains in information compression, including the ability to capture images and video in a way that is essentially resolution-less. Where the output can be rendered at any desired resolution (this already exists in various forms, which I'll show below), and even any desired frame rate.

Years ago, there was massive interest in fractal compression. The challenge was and still is, how to efficiently find features and structure which can be coded into the generative system such that the original can be accurately reconstructed. RAW images are a top-down capture of an image: brute force uncompressed RGB pixels in a Bayer array format. It's massively inefficient, and was originally used to offload de-Bayering and other processing from in-camera to desktop computers for stills. That was and still is a good strategy for stills, however for video it's wasteful and expensive because of storage requirements. That's why most ARRI footage is captured in ProRes vs. ARRIRAW. 10- or 12-bit log-encoded DCT compressed footage is visually lossless for almost all uses (the exception being VFX/green-/blue-screen where every little bit helps with compositing). Still photography could use a major boost in efficiency by using H.265 algorithms along with log-encoding (10 or more bits). There is a proposed JPG replacement based on H.265 I-frames.

DCT compression is also a top down method which more efficiently captures the original information. An image is broken into macro blocks in the spatial domain, which are further broken down into constituent spectral elements in the frequency domain. The DCT process computes the contribution coefficients for all available frequencies (see the link- how this works is pretty cool). Then the coefficients are quantized (bits are thrown away), and the remaining integer coefficients are further zeroed and discarded below a threshold and the rest further compressed with arithmetic coding. DCT compression is the foundation of almost all modern, commercially used still and video formats. Red uses wavelet compression for RAW, which is suitable for light levels of compression before DCT becomes a better choice at higher levels of compression, and massively more efficient for interframe motion compression (IPB vs. ALL-I).

Which leads us to motion compression. All modern interframe motion compression used commercially is based on the DCT and macroblock transforms with motion vectors and various forms of predictions and transforms between keyframes (I-frames), which are compressed in the same way as a JPEG still. See h.264 and h.265. This is where things start to get interesting. We're basically taking textured rectangles and moving them around to massively reduce the data rate.

Which leads us to 3D computer graphics. 3D computer graphics is a generative system. We've modeled the scene with geometry: points, triangles, quads, quadratic & cubic curved surfaces, and texture maps, bump maps, specular and light and shadow maps (there are many more). Once accurately modeled, we can generate an image from any point of view, with no additional data requirements, 100% computation alone. Now we can make the system interactive, in real-time with sufficient hardware, e.g. video games.

Which leads us to simulations. In 2017 we are very close to photo-realistic rendering of human beings, including skin and hair: http://www.screenage.com.au/real-or-fake/

Given the rapid advances in GPU computing, it won't be long before this quality is possible in real-time. This includes computing the physics for hair, muscle, skin, fluids, air, and all motion and collisions. This is where virtual reality is heading. This is also why physicists and philosophers are now pondering whether our reality is actually a simulation! Quoting Elon Musk: https://www.theguardian.com/technology/2016/oct/11/simulated-world-elon-musk-the-matrix

Quote

“Forty years ago we had Pong – two rectangles and a dot. That’s where we were. Now 40 years later, we have photorealistic, 3D simulations with millions of people playing simultaneously and it’s getting better every year. And soon we’ll have virtual reality, we’ll have augmented reality,” said Musk. “If you assume any rate of improvement at all, then the games will become indistinguishable from reality.”

A reality simulator is the ultimate generative system. Whatever our reality is, it is a generative, emergent system. And again, when you study how DNA and DNA replication works to create living beings, you'll see what is possible with highly efficient compression by nature itself.

How does all this translate into video compression progress in 2017? Now that we understand what is possible, we need to find ways to convert pixel sequences (video) into features, via feature extraction. Using artificial intelligence, including machine learning, is a valid method to help humans figure out these systems. Current machine learning systems work by searching an N-dimensional state space and finding local minima (solutions). In 3D this would look like a bumpy surface where the answer(s) are deep indentations (like poking a rubber sheet). Systems are 'solved' when the input-output is generalized, meaning good answers are provided with new input the system has never seen before. This is really very basic artificial intelligence, there's much more to be discovered. The general idea, looking back at 3D simulations, is to extract features (resolution-less vectors and curves) and generalized multi-spectral textures (which can be recreated using generative algorithms), so that video can be massively compressed, then played back by rendering the sequence at any desired resolution and any desired frame rate!

I can even tell you how this can be implemented using the concepts from this discussion. Once we have photo-realistic virtual reality and more advanced artificial intelligence, a camera of the future can analyze a real world scene, then reconstruct said scene in virtual reality using the VR database. For playback, the scene will look just like the original, can be viewed at any desired resolution, and even cooler, can be viewed in stereoscopic 3D, and since it's a simulation, can be viewed from any angle, and even physically interacted with!

It's already possible to generate realistic synthetic images from machine learning using just a text description! https://github.com/paarthneekhara/text-to-image.

https://github.com/phillipi/pix2pix

http://fastml.com/deep-nets-generating-stuff/ (DMT simulations )

http://nightmare.mit.edu/

AI creating motion from static images, early results:

https://www.theverge.com/2016/9/12/12886698/machine-learning-video-image-prediction-mit

https://motherboard.vice.com/en_us/article/d7ykzy/researchers-taught-a-machine-how-to-generate-the-next-frames-in-a-video

[meanwhile]

58 minutes ago, jcs said:

Have you noticed how @HockeyFan12 has disagreed with me politely in this thread, and we've gone back in forth in a friendly manner as we work through differences in ideas and perceptions, for the benefit of the community?The reason you reverted to ad hominem is because you don't have a background in mathematics, computer graphics, simulations, artificial intelligence, biology, genetics, and machine learning?

That's where I'm coming from with these predictions: https://www.linkedin.com/in/jcschultz/. 

I *do* have a background in exactly those subjects. Seriously. 

And, no, your work history on your linkedin page does not show any evidence of competence in anything other than basic C programming. And while I have't treated you in a way that flatters your ego, I have been perfectly polite. And, no, I haven't used any ad homs. 

Here's the biggest point of all, which I was holding back to spare you embarassment: raw is a specialized recording media, not one for transmission - that's what jpeg is for. You make raws so you can make jpegs of different qualities, fiddle with the image etc. The people whose work you have grossly misunderstood are claiming it as a possible replacement for jpeg. So where on earth did you get the idea that it could replace raw??? The point of raw is that it is lossless and includes more data than the eye necessarily needs to see. The work you so grossly mis-understood is a form of lossy compression. It is not, by definition, a potential raw replacement. 

For you not to understand this shows you not only don't understand the technology you are talking about but that you don't understand what raw is. On a video forum where you have made almost 2000 posts... 

(And no, I don't feel like sharing my real world identity with someone who, to say, the least, seems like a compulsive balloon juice drinker.)

[jcs]

40 minutes ago, meanwhile said:

I *do* have a background in exactly those subjects. Seriously. 

And, no, your work history on your linkedin page does not show any evidence of competence in anything other than basic C programming. And while I have't treated you in a way that flatters your ego, I have been perfectly polite. And, no, I haven't used any ad homs. 

Here's the biggest point of all, which I was holding back to spare you embarassment: raw is a specialized recording media, not one for transmission - that's what jpeg is for. You make raws so you can make jpegs of different qualities, fiddle with the image etc. The people whose work you have grossly misunderstood are claiming it as a possible replacement for jpeg. So where on earth did you get the idea that it could replace raw??? The point of raw is that it is lossless and includes more data than the eye necessarily needs to see. The work you so grossly mis-understood is a form of lossy compression. It is not, by definition, a potential raw replacement. 

For you not to understand this shows you not only don't understand the technology you are talking about but that you don't understand what raw is. On a video forum where you have made almost 2000 posts... 

(And no, I don't feel like sharing my real world identity with someone who, to say, the least, seems like a compulsive balloon juice drinker.)

Haha thanks for the laugh! You should try balloon juice, it has electrolytes!

I think I know how Luke Wilson's character felt  

[jcs]

Generative compression isn't a replacement for RAW, since ultimately RAW is a finite resolution capture. So RAW loses both in being fat and inefficient with additional inherent flaws due to the Bayer sampling, and more importantly is severely limited in quality as max resolution is fixed. What generative compression will do is provide both vastly smaller files and superior image quality at the same time, as the video will be rendered to the resolution of the display device, in the same way a video game is rendered to the display. Upgrade from a 1080p monitor to 4K or even 8K, and your old content has just improved along with your new display device, for free.

This is state of the art compression, and we're only getting started! https://arxiv.org/pdf/1703.01467.pdf

[UncleBobsPhotography]

6 minutes ago, jcs said:

Generative compression isn't a replacement for RAW, since ultimately RAW is a finite resolution capture. So RAW loses both in being fat and inefficient with additional inherent flaws due to the Bayer sampling, and more importantly is severely limited in quality as max resolution is fixed. What generative compression will do is provide both vastly smaller files and image quality at the same time, as the video will be rendered to the resolution of the display device, in the same way a video game is rendered to the display. Upgrade from a 1080p monitor to 4K or even 8K, and your old content has just improved along with your new display device, for free.

This is start of the art compression, and we're only getting started! 

https://arxiv.org/pdf/1703.01467.pdf

 

But the original RAW file will still contain more information than a generative compressed video file since it's a non-reversible process, and the RAW-file will therefore give more opportunities while editing.

[jcs]

43 minutes ago, UncleBobsPhotography said:

But the original RAW file will still contain more information than a generative compressed video file since it's a non-reversible process, and the RAW-file will therefore give more opportunities while editing.

Have you edited 5D3 1080p 14-bit RAW or Red RAW vs. 1080p 10-bit DCT compressed log? The only 10-bit log camera I have access to is the C300 II (also does 12-bit RGB log). In my experience, the DCT compressed 10- and 12-bit log files have actually more post editing flexibility due to the extra DR of the C300 II. I've yet to see compression cause visible issues. Have you had similar experience between RAW and 10-bit log DCT (ProRes, XAVC, XF-AVC etc.)?

So in the case of generative compression, which is lossy on the one hand (not an exact copy), it would edit the same as 10+ bit log DCT (ProRes, XAVC, XF-AVC etc.). Now here's where it gets interesting. Normally we'd opt for RAW to get every last bit of detail from the camera for VFX, especially for chroma key. Not only will generative compression not suffer from macroblock artifacts, it can be rendered at any desired resolution. It's like scanning a bitmap shape and having it converted to vector art. The vector art can then be transformed and rendered at any scale without artifacts. So even for VFX, generative compression will be much better than RAW. Maybe generative transformation is a better term, since it's effectively an improved representation (even though it's not a bit exact copy).

[meanwhile]

10 hours ago, jcs said:

Generative compression isn't a replacement for RAW, since ultimately RAW is a finite resolution capture. So RAW loses both in being fat and inefficient with additional inherent flaws due to the Bayer sampling, and more importantly is severely limited in quality as max resolution is fixed. What generative compression will do is provide both vastly smaller files and superior image quality at the same time, as the video will be rendered to the resolution of the display device

This is, of course, completely gaga. Yes, you can synthesise more levels of detail. But they are a lie and no one wants them. More than that, no one has worked out how to synthesize computer graphics that are convincing on this level. The idea of a Canon 5Dvi actually generating this code is so nutty that you should be wearing an anti-squirrel hat for your own safety.

Two more things about ML and cameras -

1. ML algorithms generally take a long time to run. And the more complex the task and the better the required result, the more time they need - and generally this relationship is a steep geometrical curve. So even if this tech existed, which it doesn't, why the devil would you try to run it at a high frame rate in a portable battery powered device? It's like a man who fell off a bridge in Paris. In. Seine.

2. Camera sensors hate heat - because it equals noise. The best IQ in any reasonable size stills camera comes from the Sigma Foveons and part of the reason is they are designed as radiators and screen resolution etc are compromised to keep heat down. Running a complex ML algorithm at video frame rates would be an unprecedented processor load, meaning unprecedented amounts of heat from the CPU. So even if the tech existed, why would you do it??? Seinely - sorry, sanely - you wouldn't. You'd take nice raws and then compress using this tech on your PC.

..Honestly, "balloon juice" is being kind.

[jcs]

1 hour ago, meanwhile said:

This is, of course, completely gaga. Yes, you can synthesise more levels of detail. But they are a lie and no one wants them. More than that, no one has worked out how to synthesize computer graphics that are convincing on this level. The idea of a Canon 5Dvi actually generating this code is so nutty that you should be wearing an anti-squirrel hat for your own safety.

Two more things about ML and cameras -

1. ML algorithms generally take a long time to run. And the more complex the task and the better the required result, the more time they need - and generally this relationship is a steep geometrical curve. So even if this tech existed, which it doesn't, why the devil would you try to run it at a high frame rate in a portable battery powered device? It's like a man who fell off a bridge in Paris. In. Seine.

2. Camera sensors hate heat - because it equals noise. The best IQ in any reasonable size stills camera comes from the Sigma Foveons and part of the reason is they are designed as radiators and screen resolution etc are compromised to keep heat down. Running a complex ML algorithm at video frame rates would be an unprecedented processor load, meaning unprecedented amounts of heat from the CPU. So even if the tech existed, why would you do it??? Seinely - sorry, sanely - you wouldn't. You'd take nice raws and then compress using this tech on your PC.

..Honestly, "balloon juice" is being kind.

[Snowfun]

Your appeal to Baer is demonstrably nonsense. Rather than simply googling for something with apparent (albeit dubious) relevance, spend a moment thinking about the quote. "All" ideas? Of course, some ideas which eventually become scientific orthodoxy (the Higgs boson might be an example) enjoy controversy and disagreement when initially proposed and only acquire acceptance once evidence is produced and analysed. I suppose the "new idea" to replace film with digital might also be a more relevant example. But "all"? I think not. An example: "the earth is flat" was once a "new idea" too...

[jcs]

35 minutes ago, Snowfun said:

Your appeal to Baer is demonstrably nonsense. Rather than simply googling for something with apparent (albeit dubious) relevance, spend a moment thinking about the quote. "All" ideas? Of course, some ideas which eventually become scientific orthodoxy (the Higgs boson might be an example) enjoy controversy and disagreement when initially proposed and only acquire acceptance once evidence is produced and analysed. I suppose the "new idea" to replace film with digital might also be a more relevant example. But "all"? I think not. An example: "the earth is flat" was once a "new idea" too...

[Emanuel]

Myth... *cough cough* Really?

From 9.4K wide sample shot on the newest Light L16:

[meanwhile]

4 hours ago, Snowfun said:

Your appeal to Baer is demonstrably nonsense. Rather than simply googling for something with apparent (albeit dubious) relevance, spend a moment thinking about the quote. "All" ideas? Of course, some ideas which eventually become scientific orthodoxy (the Higgs boson might be an example) enjoy controversy and disagreement when initially proposed and only acquire acceptance once evidence is produced and analysed. I suppose the "new idea" to replace film with digital might also be a more relevant example. But "all"? I think not. An example: "the earth is flat" was once a "new idea" too...

Or in other word: Yes, they laughed at Columbus. But they laughed at Bozo The Clown, the perpetual motion machine freaks, and the Hollow Earthers too.

...An appeal to Baer is just squirrel taunting. A lot of ideas are laughed at because they are not just wrong but hilariously wrong. And the idea of replacing raw with one of the most computationally expensive forms of "compression" imaginable is one of these.

[jcs]

58 minutes ago, meanwhile said:

Or in other word: Yes, they laughed at Columbus. But they laughed at Bozo The Clown, the perpetual motion machine freaks, and the Hollow Earthers too.

...An appeal to Baer is just squirrel taunting. A lot of ideas are laughed at because they are not just wrong but hilariously wrong. And the idea of replacing raw with one of the most computationally expensive forms of "compression" imaginable is one of these.

[meanwhile]

Being open-minded means being willing to consider ideas without prejudice on their merits. It does not mean "Give yourself a lobotomy - it's FUN!"

[jcs]

1 hour ago, meanwhile said:

Being open-minded means being willing to consider ideas without prejudice on their merits. It does not mean "Give yourself a lobotomy - it's FUN!"

 

[Emanuel]

7 hours ago, Emanuel said:

Myth... *cough cough* Really?

From 9.4K wide sample shot on the newest Light L16:

Sample fixed now -- original is 9388 x 7019:

[jcs]

16 minutes ago, Emanuel said:

Sample fixed now -- original is 9388 x 7019:

The math & science is sound- we're just getting started with computational cameras; it will get much better (already looking decent from your example).

[ND64]

On 7/26/2017 at 11:06 AM, Mattias Burling said:

85inch is pretty tiny compared to a movie screen. And the content has been ready for that since the 20ths.

2k falls apart on 85 inch. Even blueray quality 2k. 

[Mattias Burling]

2K looks nice on 20m.

On another note,

TV broadcasters have always been more interested in color depth and dynamic range In HD over 4K.

Among many cinematographers and still photographers its pretty well known that good lighting can make an image appear way sharper than resolution. Resolution is only good for lab tests.

Same with those crazy lens sharts. An image from a lens with a technically low score can be sharper and more alive than an image from a high scoring lens.

Specs still don't mean jack.