Color detail issues in Fujifilm video files

[Attila Bakos]

I found something strange when I was looking at the chromaticity channels of Fujifilm video files and did some search online, and to my surprise noone is talking about this, so I decided to make a video about it. If you're a Fuji user, you might be interested.

 

[tupp]

Very nice presentation!

The "washed-out" colors from the Fuji files perfectly demonstrate a fundamental imaging principle:  Color depth is a direct function of resolution and bit depth.

Here is the color depth formula for RGB digital systems: COLOR DEPTH = (RESOLUTION x BIT DEPTH)³

The Fuji color smoothing (or any other chroma sub-sampling method) reduces the resolution of the chroma channels, and, thus, reduces the color depth -- even though the bit depth remains unchanged!

The video mentions that Fuji files yield a UHD luma channel and HD chroma channels.  However, it certainly appears that the Fuji chroma smoothing reduces the effective chroma resolution to significantly less than HD, which makes your comparisons a dramatic example of how resolution affects color depth.

By the way, the reason that the Fuji jpeg stills have better color depth (even with the Fuji chroma smoothing) is likely due to the fact that they have a higher resolution.

Thanks for these comparisons!

 

 

[Xavier Plagaro Mussard]

Wow, that is one discovery! Shows that specs are one thing, reality another.

[Attila Bakos]

47 minutes ago, tupp said:

However, it certainly appears that the Fuji chroma smoothing reduces the effective chroma resolution to significantly less than HD

 

There must be some clever algorithm behind this as it seems to have proper 1080p resolution around well defined edges, but everywhere else it's clearly less than that.

[Django]

Isn't this similar to what Fuji still users refer to as the "watercolor" or "worm" effect , especially present on foliage?

If so it basically has to do with the X-Trans sensor vs Bayer sensors.

I left Fuji system years ago (and partly because of such artifacts and color issues) but search for "fuji watercolor effect" and you'll find tons of talk about it.. 

 

[Attila Bakos]

13 minutes ago, Django said:

Isn't this similar to what Fuji still users refer to as the "watercolor" or "worm" effect , especially present on foliage?

If so it basically has to do with the X-Trans sensor vs Bayer sensors.

I left Fuji system years ago (and partly because of such artifacts and color issues) but search for "fuji watercolor effect" and you'll find tons of talk about it.. 

 

This is not a demosaicing problem, it seems like some sort of temporal noise reduction applied to the chroma channels, as the first frame was always worse. I skipped the first few frames in each clip because of that.
H.264 and H.265 didn't make a difference. Long-GOP was marginally better in some situations, but generally it's just as bad as All-I.

[Django]

hmmm, no idea then but what I do remember was being shocked at how poorly Fuji files handled the red channel vs Canon. I also had a 5D3 at that time and would do side by sides with my XT2 and it was night & day. It would drive me nuts!

[kye]

@Attila Bakos - great presentation!

It's nothing to do with resolution @tupp, otherwise the RAW files would be impacted too, it's processing.

I'm not convinced it's chroma NR either, as normally NR is just blurring.  Could it just be extreme compression?  If it gave the Y channel most of the bitrate maybe that's what happens to the colour channels?

[Attila Bakos]

55 minutes ago, kye said:

@Attila Bakos - great presentation!

It's nothing to do with resolution @tupp, otherwise the RAW files would be impacted too, it's processing.

I'm not convinced it's chroma NR either, as normally NR is just blurring.  Could it just be extreme compression?  If it gave the Y channel most of the bitrate maybe that's what happens to the colour channels?

Thanks! Double checked my files, it's hard to say. The EOS R has way more noise in the chroma channels, it looks like something that you'd expect from a log recording, whereas the Fuji is unnaturally noise free. Can extreme compression have this effect? I'm not sure about that. Also, why would they assign sooo low bitrate from the 400Mbps to the chroma channels? It doesn't make any sense as above 100 it's hard to notice the difference even in luma, and as I have shown in the video, you can have nice luma+chroma even with 100Mbps.

[kye]

30 minutes ago, Attila Bakos said:

Thanks! Double checked my files, it's hard to say. The EOS R has way more noise in the chroma channels, it looks like something that you'd expect from a log recording, whereas the Fuji is unnaturally noise free. Can extreme compression have this effect? I'm not sure about that. Also, why would they assign sooo low bitrate from the 400Mbps to the chroma channels? It doesn't make any sense as above 100 it's hard to notice the difference even in luma, and as I have shown in the video, you can have nice luma+chroma even with 100Mbps.

It depends on how they've gone about making the chroma channel look like that.  Maybe they blur the crap out of it and the compression algorithm just goes "ooh, no detail...  wheeeee!" and auto-allocates it stuff all bitrate.

You do have to look closely at the images to notice (although once you see it you can't un-see it).

[Attila Bakos]

1 hour ago, kye said:

It depends on how they've gone about making the chroma channel look like that.  Maybe they blur the crap out of it and the compression algorithm just goes "ooh, no detail...  wheeeee!" and auto-allocates it stuff all bitrate.

You do have to look closely at the images to notice (although once you see it you can't un-see it).

Even if they blur the crap out of it, they do it in a smart way, I'm not sure if it's visible in the video but the houses and vehicles have good chroma details in the Fuji, it's the foliage that suffers a lot, where you're not expecting too much color difference.

[kye]

1 hour ago, Attila Bakos said:

Even if they blur the crap out of it, they do it in a smart way, I'm not sure if it's visible in the video but the houses and vehicles have good chroma details in the Fuji, it's the foliage that suffers a lot, where you're not expecting too much color difference.

There are all sorts of image processing algorithms they could be using.  Whatever it is though, it's not high quality!

It's such a pity as Fuji have such a great reputation for their colour science.

[Attila Bakos]

42 minutes ago, kye said:

There are all sorts of image processing algorithms they could be using.  Whatever it is though, it's not high quality!

It's such a pity as Fuji have such a great reputation for their colour science.

People like to mention smooth skin tones when it comes to Fuji JPEGs. I prefer to have all the details and do my own smoothing, but for many people this phenomenon is actually a plus 🙂
I was searching online a lot after I discovered this, I couldn't believe that no one is talking about this. I think I only found one old topic from dpreview where someone mentioned chroma smoothing happening in Fuji JPEGs, but I can't find it now. So we are quite alone here 😄

[androidlad]

Nice video, inquisitive as always.

This is mostly due to the nature of X-Trans, there are far more green photosites than red and blue, compared to traditional Bayer sensors, hence the less detailed Cr channel. The benefit is that it naturally reduces moire and it has a bit cleaner luminance detail (Y channel).

You won't notice this on videos shot on Fuji GFX cameras which use Bayer sensor.

[Django]

So looks like I was right, it is because of Xtrans vs Bayer sensor.. again google « Fuji watercolour effect » you’ll find plenty of talk about it..

[Attila Bakos]

1 hour ago, androidlad said:

Nice video, inquisitive as always.

This is mostly due to the nature of X-Trans, there are far more green photosites than red and blue, compared to traditional Bayer sensors, hence the less detailed Cr channel. The benefit is that it naturally reduces moire and it has a bit cleaner luminance detail (Y channel).

You won't notice this on videos shot on Fuji GFX cameras which use Bayer sensor.

I believe this has nothing to do with demosaicing, all the color information that's missing from the videos/jpegs are there in the x-trans raw files. I can show it tomorrow if necessary.

Edit: looking at Django's comment it's necessary 🙂

[Django]

@Attila Bakos Not really necessary. This article breaks it down, it has to do with Xtrans & Fuji’s heavy chroma NR jpeg engine :

https://petapixel.com/2017/01/27/x-trans-promise-problem/

[Attila Bakos]

11 minutes ago, Django said:

@Attila Bakos Not really necessary. This article breaks it down, it has to do with Xtrans & Fuji’s heavy chroma NR jpeg engine :

https://petapixel.com/2017/01/27/x-trans-promise-problem/

But this proves my point. It's the chroma NR. The reason they apply it might be related to X-Trans but that's all, X-Trans alone won't cause what you see in my video. Easy to prove, I might do it later when I get home.

[Django]

All I’m saying to you is that it’s a well documented issue and that article details exactly what’s going on, the reason why and potential solutions Fuji has yet to adopt except by going Bayer on GFX series:

The reason that FujiFilm hasn’t fixed the Waxy Skin-Tone problem after all these years and three camera generations is simply that they can’t—not without a significant, breakthrough advancement of their algorithms. They made a proverbial deal with the devil with their immune-to-moiré claims for the X-Trans CFA. If they get rid of the moiré, people will complain about waxy skin tones, and if they get rid of the waxy skin tones, people will complain about moiré (but, hey, they never made marketing claims about not making people look like wax figures or wooden dolls, so it’s no surprise which problem they’ve prioritized.) 

Certainly, more sophisticated moiré suppression algorithms could do better at sparing skin and faces where the color smoothing is most objectionable (but could they ever recognize chicken skin, I wonder?). Will Fuji ever invest any time or money in that kind of optimization? Especially with reviewers constantly praising their JPEG engine (from a comfortable distance and without wearing their spectacles)?

In the future, I sincerely hope that FujiFilm stops producing sensors with the X-Trans CFA. I am convinced that the X-Trans CFA causes more problems (many) than it solves (none) and FujiFilm could have well known this from computer simulations before ever manifesting it physically. If the GFX 50s is any indication, then the next generation of X-series cameras may indeed utilize a Bayer CFA.

In the meantime, a firmware update to give users the option to customize the chroma NR/moiré removal strength ramp ourselves in camera (like some other manufacturers do with their High ISO NR customization) could alleviate the problem for users who consider the lifelike rendition of human faces more important than complete freedom from moiré. I challenge FujiFilm to offer a solution. 

[Django]

By the way no need to prove that RAW files hold the missing colour information as the article already states that:

On the bright side, it has been shown here that much more color information can be extracted from the raw sensor data than FujiFilm’s in-camera processing is currently capable of. 

The alternative is to shoot RAW. Which is fine, but demosaicking X-Trans files is less efficient than demosaicking Bayer files—anyone who as tried it knows this translates to a much slower workflow. Plus you lose all of that “color science” too, because, sadly (shamefully, in my opinion), FujiFilm does not publish color profiles for their sensors nor embed the color matrices in the RAW files as some other manufacturers do.

Unfortunately, RAW video hasn’t yet made it to Fuji cameras so it seems likely you are stuck with this issue as you can’t bypass the in-cam heavy chroma NR in the compressed h264/h265.