Let's discuss computational photography?

[Emanuel]

 

Specifically for the iPhone 11 Pro, Phil Schiller previewed an upcoming Deep Fusion feature that analyzes 9 photos (including one long-exposure shot) taken in sequence. The device’s neural engine then analyzes the collection to create an optimal end photo that borrows the best elements of each image. Schiller said it was “computational photography mad science.” Deep Fusion is coming in the fall.

https://www.theverge.com/2019/9/10/20850694/apple-iphone-11-pro-max-camera-specs-features-triple-three-slofies

Since iPhone 7 Plus by Apple...

How game changer is such contribution?

I find it pretty revolutionary to be quite straightforward (E : -)

[gethin]

3 hours ago, Emanuel said:

 

How game changer is such contribution?

massive.  prediction: the sort of processing I do on real estate pics will be pretty well dead in 5-10 years time.  They won't be as good, but they will be "good enough".  I can imagine the new image processing software will just have  bunch of looks, and ai will do the rest. This is the reason I'm trying to get out of the game now.  It's bound to come to video, too. 

 

[kye]

I remember reading about this when it was first introduced, and maybe I'm remembering incorrectly but I think the iPhone 5 had some.  My iPhone 6 Plus definitely had some - you would take a shot and then take a burst, and if you compared the single shot it had far less ISO noise than each frame in the burst.  It lowered the ISO noise by taking a burst and combining them to average-out the ISO noise.

People have criticised Apple for not competing in the megapixel wars but I remember reading that Apple were putting the horsepower into the image processor instead of the sensor because they were looking at photography in a new way.

The original way is: one exposure from the sensor -> save straight to storage (eg, film)
The old way is: one exposure from the sensor -> colour and image processing (colour science, lens distortion compensation) -> save to storage
The new way is: many exposures from many sensors -> colour and image processing of each exposure (like above) -> processing many images together to create a single image -> save
The future will be: many exposures from many sensors -> colour and image processing of each exposure (like above) -> processing many images together to create something more than a single image -> save. (this may be combining images into a 3D environment, a VR output file, a file that lets you choose focus point and DoF in post, etc)

The processing may also do things like combine multiple images together to find the best moment from a facial expression perspective ("peak smile" or "eyes open"), or more sophisticated would be choosing separate exposures and combining them together to pick the best frame for each person in a group shot so everyone is at peak-smile and no-one has their eyes closed.

Cameras that learn your style from giving you many options and you choosing one will come, this will be a precursor to you not needing to hit the shutter button or even engage the camera - once they can analyse a scene they will know what a touching moment looks like and will just be watching the whole time and will just save (or flag) the best bits, creating little sequences for you.  Pair the Apple Glasses with the Apple Watch and your iPhone and the watch will know when your heart rate is up and hear your vocal intonations to know if this exciting moment is the applause for your kid in the school play or a car accident, the Apple Glasses will be sending high-speed image sequences, and your iPhone will be processing and storing the whole lot.

It will create magical memories from your life, it will save all the steps in how you made that recipe so you can make it again, it will create BTS videos, it will automatically log who you met with and what you spoke about so you can network and manage business and sales opportunities.  It will be monitoring what you buy, it will be analysing and categorising and diagnosing you, and it will be talking to Apple about what you might want to buy next.

It will be magical, people will queue up around huge city blocks for it, and the privacy warriors will go absolutely bananas.

[JurijTurnsek]

I don't want my face enhanced in any way.

I fail to see how this is any better that super quick HDR with more merged shots as the processing power increases (with "beautification" applied at the end? That Apple presentation photo is a very good example at how they can blow up any little feature to seem like a whole new world.

Someone please tell me who decides what the hell is "the best part of a photo"?

[kye]

1 hour ago, JurijTurnsek said:

I don't want my face enhanced in any way.

I fail to see how this is any better that super quick HDR with more merged shots as the processing power increases (with "beautification" applied at the end? That Apple presentation photo is a very good example at how they can blow up any little feature to seem like a whole new world.

Someone please tell me who decides what the hell is "the best part of a photo"?

It's not for everyone, but if you're not familiar with what computation can actually achieve then you might be surprised.

In terms of what "the best part of a photo" is, you'd be amazed at how things like facial expressions can be universal, how complex things can be analysed with the right mathematics, and how improvement over time can make huge leaps in a technology.  It's a whole new world and it will be surprising to many how far this technology will come.

[sanveer]

Somehow the Quad Pixel idea (and especially that Along With the multi frame HDR) seems like a better idea than regular multi frame HDR (or even multi frame low light). By shrinking the pixel size, we get the advantage if both wider dynamic range, as well as better low light, and most importantly, we get none of the regular motion blur associated work combining multiple frames. I am guessing that the present Snapdragon processors aren't equipped to handle the full potential of the Quad Bayer pixel sensors, atleast not for the moment. 

[Mokara]

Oh. I see. So Apple have "invented" bracketing now?

[Geoff CB]

I use Topaz Gigapizel every single day at work. I constantly have to pull images for ads and for preshows that I make and being able to upscale the image multiple times and get a better result from even basic material saves me I would say an average of an hour each week. 

You better believe that I would use an AI upscaler in my editing program, or denoiser, or software that could refocus a defocused shot.

It will change the entire world of media. 

[hoodlum]

9 hours ago, JurijTurnsek said:

I don't want my face enhanced in any way.

I fail to see how this is any better that super quick HDR with more merged shots as the processing power increases (with "beautification" applied at the end? That Apple presentation photo is a very good example at how they can blow up any little feature to seem like a whole new world.

Someone please tell me who decides what the hell is "the best part of a photo"?

The difference is that Google/Apple is selectively merging regions of each image based on what has moved from frame to frame.  If you just merge images then you will capture the blur in every movement.  This will be getting more complex over time and currently some of this cannot be handled with post-processing and may never be able to.

[Mokara]

43 minutes ago, hoodlum said:

The difference is that Google/Apple is selectively merging regions of each image based on what has moved from frame to frame.  If you just merge images then you will capture the blur in every movement.  This will be getting more complex over time and currently some of this cannot be handled with post-processing and may never be able to.

I don't see why not, since what they are effectively doing is the same thing, just in the camera.

[Emanuel]

3 hours ago, Mokara said:

Oh. I see. So Apple have "invented" bracketing now?

No, Apple has introduced their dual-camera system on their phones since their 7 Plus model version which has opened the door to reinvent a new usage for bracketing applied to computational photography.

[Mokara]

So, they invented bracketing. I see.

[Geoff CB]

4 hours ago, Mokara said:

So, they invented bracketing. I see.

You can repeat it all you like, doesn't make you correct. 

Google has "NIght Signt" in their phones. It takes multiple images in a row, analyses them to reduce noise and sharpen them into a detailed final image. It's fantastic tech and extremely useful for non-action photography. Apple is just doing something similar but taking it a step further.

[JurijTurnsek]

Unless Apple implements a ToF sensor in the front, they are just doing "smart/zone" bracketing, which is wholly in the Android domain of possibility. None of the discussed samples feature fast movement, just landscapes and portraits and Dpreview is not all that impressed at the pixel level.

Hopefully, Android OEMs push for bigger sensors at the same time, since apple is very much afraid of thicker phones. Bracketing, fast readout and bigger sensors would be really push mobile photography further.

[sanveer]

Panasonic cameras have that new Composite image function where it merges many photos into a single images, showing movement with its main subject within the frame. The issue with that is thaybrje background has to be extremely still and anything moving in the background makes image look extremely strange. There appears no reason why smartphones can't do it, though I would call Apple's bluff on it, for the moment. I expect Goggle to come out with such an algorith first, mostly because its surveillance/spying abilities for technology are second to none

 

[Mokara]

19 hours ago, Geoff CB said:

You can repeat it all you like, doesn't make you correct. 

Google has "NIght Signt" in their phones. It takes multiple images in a row, analyses them to reduce noise and sharpen them into a detailed final image. It's fantastic tech and extremely useful for non-action photography. Apple is just doing something similar but taking it a step further.

Lol....that is exactly what bracketing is. It can be used for all sorts of things, and works, provided there is not excessive motion in the picture. The point is, Apple did not invent it, it has been around for as long as digital cameras have been around.

The same principle is used in focus stacking for example.

[Emanuel]

Nevertheless, thanks to you now we know the wheel was invented : D

Shame on Henry Ford too who died to think he had invented the concept "any color so long as it is black" : P

[heart0less]

What really excites me about the whole computational thing is the fact that it clearly shows major breakthroughs are no longer dependent on optics, sensors, but on processing power and software.

I recently fooled a Camera app into thinking that my dearly beloved Redmi Note 4 is in fact a Pixel 3.

The results truly amazed me.

 

Just like gethin said, those photos may not be the best in terms of color rendering, ability to resolve details (not to be confused with sharpness), but they're good enough, especially for not tech-savvy people.

I'm just worried that all my gear might become obsolete in 5 years time - I guess it's called 'progress' and 'moving on'.
( :

[Emanuel]

https://noamkroll.com/why-iphones-are-now-the-inevitable-future-of-independent-filmmaking/

 

[Snowbro]

Eventually people (actors) will license their likeability/3d model out and not actually bother to act in a movie. Look at what they did with Paul Walker after he was dead.