RAW Video on a Smartphone

[sanveer]

 

 

This is Pretty Neat. Unfortunately, since the Sony IMX377 and other 1/2.3" sensors, most smartphone sensors have abandoned 12-bit, for photo or video (Staggered HDR sensors do 12-bit HDR for photos instead). So, this is limited to 10-bit video for now. But, the video quality looks pretty stellar. Unprocessed and I am guessing the highlight and shadow recovery maybe pretty decent. It apparently does 10-bit 4-2-2 (most smartphones that do 10-bit video, have 4-2-0).

(The new Snapdragon and MediaTek SoCs talk about 18-bit imaging, which I am guessing may be some crazy HDR, upressing bit-depth, though that may solely be limited to photos).

This RAW video does look pretty neat, and doesn't look over sharpened or over processed like most smartphone footage. 

 

[Emanuel]

Here's another one, not from LG now but an old Samsung capture device... What a feast!

No ISP processing crap! :- )

[billdoubleu]

Here's the link to GitHub for the motioncam app:

https://github.com/mirsadm/motioncam/releases

I am going to see if I can get this to work on my Pixel 3 XL tonight. Here's hoping, but google uses slow storage.

[billdoubleu]

The Pixel 3 XL is dropping frames like nobody's business! Hopefully someone with a powerful phone can have fun with this.

[slonick81]

Wow

Tested on my Poco X3. TLDR: barely functional but promising.

Settings: anything aside RAW10<->RAW16 show little effect on performance. "Raw video memory usage" seems to benefit from being maxed out but it's more a feeling than fact. You'll have to select "raw video" and exposure settings every time you switch from app's main screen (like switching between apps or entering settings). OIS can be activated but doesn't work in my case, thus the image is shaky. Focus control is kind of distributed strangely: 90% of slider is mapped to nearest 1m, the rest of focusing range is cramped in the tiny space at the right end. Nice for macro, I guess, not for anything else. AWB should be locked or it will affect the image otherwise.

Raw: I was able to get mostly reliable recording in all conditions at max crop (40% H/V, 2772x2082) at 24/25 fps for durations about up to 1-1,5 min. Was not testing for longer times because of huge file sizes and some quirks. I was able to record 4K-ish resolutions (4112x2082) outdoors (it's about 0°C now) but it drops frames starting from 15-20s indoors. Overheating? No framing guides for crop area - use your imagination looking at full sensor image feed. Raw images are initially written in chunks of 900 frames as zip archive somewhere in system folders (haven't found location yet). You have to manually unzip and transfer them as dngs in "manage videos" by tapping "queue" button. You can set the destination folder once at first conversion, I was not able to find this setting anywhere else, the only way to change this directory was to reinstall the app. And yeah, no sound at all yet.

Processing: transfer times are huge on my phone mostly because of USB2 transfer speed. Files are bulky, I filled 60GB of free space just with a dozen of clips. Considering buying TF card to unzip dngs and swapping it out to card reader (if projects goes well in this direction). AE2021 and Resolve 17 work well with dng sequences, Premiere 2021 refused to import. I had to rename dng sequence according to unique folder name manually because they were named with same base name - frame-#####.dng. There is some glitchy "transitional zone" between two 900-frame chunks where frames are randomly tossed to different chunks. At first I thought it was just frame dropouts but later I was able to reconstruct frame order manually at post - see image. Image quality is much better, especially at moderately high ISOs (800-1600). Compressed image degrades at this range - details are getting mushy, colors - muddy. Raw on the contrary has very manageable grain structure and highlight rolloff, lacks any sharpening. DR is better in a sense that you're getting more usable DR but it's far from being cinema camera wide - you should be careful with highlights, and the absence of any exposure assisting tools doesn't help at all.

Is it worth trying? Yes, I think. It's not the way you'd shoot something for fast turnarounds, or for a long duration. It's more an artistic experiment. The project is in a very early stage now but it's all about polishing interface and performance, adding useful features, improving general stability - the core idea is functional. I'm really happy to stumble upon this project - I haven't feel such joy and excitement since shooting photos with Nokia 808. Basically it's a 8mm raw camera you have no excuses not to carry with you all the time.

[slonick81]

Some DR tests on rare sunny winter days - raw images vs. native camera app video recording. No detail enhancements for raw (no NR, sharpening, dehazing and such), just shadows/highlights and gamma curve adjustments in ACR to get the better idea of what's going on. All shots are in native resolution, 1:1, no scaling.

I'm still not confident about exposure settings because of live feed high contrast. I tend to check the overall image with AE on and then dial manual settings to nearly same looking picture. Native video is shot in auto mode. It actually represents well in terms of contrast the image you see in live view shooting raw.

So, as you can see there is no big difference in DR (like you may expect comparing raw and "baked-in r709" footage from cinema camera). "Bridge" scene got a bit overexposed in video compared to raw, I have a feeling that it was possible to bring back overexposed building in the back without killing shadows with minor exposure shift. What makes the difference for me is highlight rolloff - it is natural and color consistent, just look at snow in front in "hotel" shot and sun spots on the top of "bridge" shot. Shadows are more manageable for me, too. Yes, there is a lot of chroma noise but it's easily controlled by native ACR/Resolve tools, luma noise is very thin and even, so you can balance between shadow details and noise suppression up to your taste.

But the most prominent difference is due to zero sharpening. Just check small details like branches, especially on contrast background. It's not DR related, but I can't avoid mentioning it.

Is raw a necessity for such result? Dunno. Maybe no postprocessing, log profile and 10bit 400Mbps h26x instead 8bit 40Mbps will give comparable results. If you have a phone that is capable of such recording modes I would love to see the compare. 

[sanveer]

Interesting screenshots. The RAW has no burnt-out highlights and the rolloff is also very ... organic. From a distance it may seem to have less detail, but, that is only unnecessary sharpening in the regular screenshot. The RAW video should look even better with some more grading and sharpening, I am guessing.

 

I noticed that, in the 2nd pic, the store signboard has absolutely no detail in the regular pic, whereas in the RAW, its visible. 

[slonick81]

On 12/11/2021 at 7:29 PM, sanveer said:

The RAW has no burnt-out highlights and the rolloff is also very ... organic. From a distance it may seem to have less detail, but, that is only unnecessary sharpening in the regular screenshot. The RAW video should look even better with some more grading and sharpening, I am guessing.

Original image is even more natural and neutral, I was trying to add some contrast to highlights and shadows to see if there any banding or other issues aside expected noise. "Processed" look is easily achievable by applying contrast, strong NR and strong unsharp mask with radius about 1,5-2px. The lens itself is somewhat soft (it's a 200$ phone) and it's impossible to degrease it every time you get it out of pocket (I clean it with dedicated cloth but it's still not ideal - sometimes you can guess by light streaks the direction of last cleaning move :)

On 12/11/2021 at 7:29 PM, sanveer said:

I noticed that, in the 2nd pic, the store signboard has absolutely no detail in the regular pic, whereas in the RAW, its visible. 

Well, I still think there is exposure shift for compressed image, it should have retained at least some details of building sign. The thing is everything goes harsh and crude in highlights really fast - color, gradations, details

 

I'm really pleased to noticed that Mirsad Makalic (the author of Motion Cam) is working hard on improving the app. A lot of issues I mentioned are addressed - you can set up you storage location any time, the story about 900-frame chunks is history - this part is completely reworked, focus slider is much more usable, RAW video buffer can be bigger and there is indicator for filling it during capture... I mean - join in, your input may be valuable, and you have chance to shape the app for your needs.

[billdoubleu]

So, I recently picked up a Pixel 5a on sale and gave the Motion Cam app a go on it. I've been having some fun and finding some fairly consistent results with the full sensor (4032X3024) @ 12FPS. I think the sensor size comes close to super 8mm sizing and the results definitely have an 8mm vibe.

The screen grab below is from a cinemaDNG sequence dropped into Resolve on a 1080p/ rec709/ linear timeline -> color space transform (which I didn't even know you could use in the free version) ARRI LogC output gamma -> ARRI Alexa LogC to rec709 LUT -> bumped up the saturation a bit and brought the shadows down with the log wheels

Be gentle; what I know of a RAW workflow has been learned in about 2 hours this week.

There's no noise reduction applied, but it seems like the downscale helped clean things up and give a more film like quality.

The app is a bit clunky, but is obviously in beta and can't be fully criticized yet. Manual focus would be amazing and there were some exposure shifts that may or may not have been my fault. If someone sold this technology with a MFT lens mount on it as a modern 8mm camera I would for sure buy one. It's been fun and I'm enjoying learning about RAW correction/ grading.

There seems to be a few people catching on to the app on YouTube. People with more powerful phones seem to be having better luck with things. Hopefully development continues with success and RED doesn't interfere!

[sanveer]

 

On 12/18/2021 at 2:33 AM, billdoubleu said:

So, I recently picked up a Pixel 5a on sale and gave the Motion Cam app a go on it. I've been having some fun and finding some fairly consistent results with the full sensor (4032X3024) @ 12FPS. I think the sensor size comes close to super 8mm sizing and the results definitely have an 8mm vibe.

Wow, the image looks great.

 

On 12/14/2021 at 5:09 PM, slonick81 said:

I'm really pleased to noticed that Mirsad Makalic (the author of Motion Cam) is working hard on improving the app. A lot of issues I mentioned are addressed - you can set up you storage location any time, the story about 900-frame chunks is history - this part is completely reworked, focus slider is much more usable, RAW video buffer can be bigger and there is indicator for filling it during capture... I mean - join in, your input may be valuable, and you have chance to shape the app for your needs.

I wanna try it. Am just a little hesitant, like with trying out the various hacks on the GH2. I'm sure thus is going to be great. I have a spare smartphone. Maybe I use it.on that one. 

[PannySVHS]

Would be nice to have a list of compatible phones. Would love some 2K+ 3:1 Raw from a Panny Lumix CM1 phone with its S16 size sensor from the RX100 series. But it has only Android 5 and 2gigs of Ram with 128gb storage to micro SD.

@sanveer Would be great to see and read your findings. Merry Christmas to you and everyone.:)

 

[PannySVHS]

How about this baddie: YN455 with mft sensor, 6gig ram and android 10. Cost still much higher than a used OG BMPCC.

[Emanuel]

I've been testing it here with a few devices... Very interesting user interface, as Spartan as I like it : )

But, the more processing power the better, I don't recommend old slow chips, it doesn't make sense, for the sake of it, the newer and top the better : )

Works like a charm, incredible tool, fairly filmmaking friendly as expected IMO ; )

I just detect a bug so far but if you reboot it you're fine : ))

I only miss faster frame rates other than 24/25/30... but you have slower range for your joy experiments so very eclectic, one of my fav apps so far < 3

@Andrew Reid what about an article about it, pal? :- )

[Emanuel]

PS:

Reboot or retart the app, I mean, not the device... It goes all of a sudden lower than its standard 4000 wide (aspect ratio is fully customizable... where else you find it, humm?) when you play and give much use to extract the best from the intuitive menus along the most recent beta version as of today.

Some control on WB other than auto mode you can block anyway wouldn't hurt either at all.

Ingenious piece of software and for free! ;- )

[sanveer]

19 hours ago, Emanuel said:

I've been testing it here with a few devices... Very interesting user interface, as Spartan as I like it : )

But, the more processing power the better, I don't recommend old slow chips, it doesn't make sense, for the sake of it, the newer and top the better : )

Works like a charm, incredible tool, fairly filmmaking friendly as expected IMO ; )

I just detect a bug so far but if you reboot it you're fine : ))

I only miss faster frame rates other than 24/25/30... but you have slower range for your joy experiments so very eclectic, one of my fav apps so far < 3

@Andrew Reid what about an article about it, pal? :- )

What device did you get to test it on? I hope they list the devices it works best on (stability, more frame rate options etc). 

Btw, I read a rumour that the new Samsung 22 Ultra is going to shoot RAW. If it shoots raw video, that would be superb. It already does 10-bit upressed to 12-bit HDR for photos, so I hope its 12-bit RAW video. Not sure why smartphone makers are already supporting RAW. If their cameras can do 10-bit 4-2-0 already, and 8k (8-bit?), I am sure they have the bandwidth and the processing power. Any even get that rubbish Camera2API on devices, with its limiting features. 

[Emanuel]

1 hour ago, sanveer said:

What device did you get to test it on? I hope they list the devices it works best on (stability, more frame rate options etc). 

Btw, I read a rumour that the new Samsung 22 Ultra is going to shoot RAW. If it shoots raw video, that would be superb. It already does 10-bit upressed to 12-bit HDR for photos, so I hope its 12-bit RAW video. Not sure why smartphone makers are already supporting RAW. If their cameras can do 10-bit 4-2-0 already, and 8k (8-bit?), I am sure they have the bandwidth and the processing power. Any even get that rubbish Camera2API on devices, with its limiting features. 

OnePlus 9 Pro.

I will try with the previous processor going along the superb ASUS Zenfone 7 Pro (both are able of 8K 30p and 4K 120fps recording).

But, I would not hold my breath too long with smartphone manufacturers...

Their motion picture sensibility is the same of someone who doesn't know where the local movie theater is : D

Their interface shooting is a truly PITA for filmmaking in 99% of cases, unfortunately.

 

- EAG

[billdoubleu]

I found some time to mess around with this video a bit more. Same screen grab as before but on a 4k timeline with Blackmagic 4k Film output gamma and I played with the colors a bit. BTW, I do not have anything close to a grading monitor. I play these videos through my Plex server on my budget TV and if it looks right I go with it. Though, I haven't looked at the video as is yet. I also overlaid the video with 16mm film grain. I'm generally not a fan of film grain for my own stuff, but, it seems to fit-the-bill here.

The colors are so much more easily manipulated here as compared to my usual G85/ GX85 CineD files. I have also learned that Davinci resolve will not export any of the frame rates I want for this video (e.g. 12, 16, 18fps). So, I am exporting 24fps and using handbrake to convert to 12fps. Handbrake also doesn't output 16 or 18fps. Any tips or tricks are welcome!

I took some 4k 60p video with the standard pixel camera app in this same scenario. I started playing around with that mess last night. The image is horrendously oversharpened and noisy. The crop is ridiculous with the OIS activated, as if it weren't bad enough with the 4k crop alone. Curiously enough the 1080p crop is the same as the 4k, so, even in 1080p they aren't sampling the full width of the sensor. You also can't do much of anything with the image in post. I made some slight contrast adjustments with curves, but anything else blows the image into orbit.

 

[Emanuel]

How many of you have used and how many of you have missed it?

 

What's the beef? No ultimate fastest phone chip...?! ; ) I am using a €700 OnePlus 9 Pro Snapdragon 888 very successfully BTW ;- )

 

25fps is not going now with the last version (8.0.7-beta2) since it says it is unsupported by my device, so I am using 24fps then.

They should likely be pretty accurate on spot but bugs look like fixed, even along a beta version.

And shutter speed is now possible to set up along ISO without any (variable and inaccurate) issues (from previous version since four days ago : )

We have now 1/48, go figure! ;- )

That is, pure and simple photography is finally possible on a smartphone! (you still have AF or how to lock it!)

 

This is plainly the best software ever made for really mobile cinematography, period! :- )

I am fairly enthusiastic on this tool indeed : X

https://github.com/mirsadm/motioncam/releases

 

This requires a bit more of attention and support from y'all without exception or pick one: you're the last elitist on earth or you don't have a smartphone? ...or yet, no, I am not missing anything else on here : D

[Andrew Reid]

Amazing image quality on the Xiaomi Mi 11 Ultra

Take a look at the full res 4K RAW frame

A crop of it below:

It has all the characteristics you'd expect from RAW on the original Blackmagic Pocket Cinema Camera but in 4K!

Noise is quite manageable up to ISO 1600.

File sizes are around 3.8-4MB per frame though, so make sure you try this on a phone with a lot of built in storage 🙂

Not tried the pixel binned or 1080p modes yet.

Also has a feature to crop the aspect ratio to a cinemascope style one, which may reduce file sizes. Didn't seem to change my end-recording though which was 4080 x 2152.

On the Mi 11 Ultra, it reported 24p as officially compatible

But also said 120p and 48p 4K may work too.

Much to explore here.

[webrunner5]

Unimaginable tech in an item that is so small.  I have no clue how people can be so smart to be able to design something like it.