My love/hate relationship with the XC10 and grading the C-Log files that I shot

[kye]

I've owned the XC10 for a few years, shot a few family travel videos with it, loved using it but always struggled with the footage it gave me, and have moved on with other cameras since which haven't given me the challenges with the footage that the XC10 gave me.  I have made attempts to edit the footage from it but got bad colour, gotten discouraged, and given up.  I'm now back to editing the footage and am determined to prevail.

This thread is my attempt to make sense of WTF is going on with XC10 C-Log footage, overcome it, and move on.  This is going to be long and complicated, so settle in and grab a drink, or go do something else (probably much more useful) with your day.  

It's not a review of the XC10, although in a sense it will go some ways to review the footage I've gotten out of it.

The context is that I shot on full-auto with this camera in available-light situations, with no control over what I shoot.  Probably most importantly is that I shoot auto-WB.  I do this for lots of reasons and even if I'm wrong for doing it, it's done, it's baked into the footage I have here, so not much I can do about that now.

The first phase of owning the camera was using it, not getting the results I wanted (in terms of colour), and thinking I didn't know enough about colour grading.  I didn't think that grading C-Log should be so difficult, considering that Canon is known for having great colour (as opposed to early Sony cameras where many people struggled in post) but I couldn't work it out.

The second phase was when I'd learned more about colour grading and I still couldn't get the results I wanted.  I was trying to grade a short video of a day-trip I took on a holiday in Italy and just couldn't get good colours.  At the time I ended up just releasing it (which means uploading it unlisted and sending it to friends and family) despite not being happy with it.

Here's the video I posted at the time:

A nice little vignette of a tiny village in Italy, shot in a couple of hours one afternoon, but oh boy did I struggle to make this video.  Despite the colours being pretty awful, this was the result of days of work.  DAYS.  After posting it I kept trying to do better.  I reached out to the colourist forums and their advice was to try for punchier contrast and more saturation, which I tried, but it always looked awful.

By far the worst shot was the one of the violinist, which was recorded in the XC10s 1080p50 mode, in mixed lighting.  Here is that shot in C-Log:

So, you apply a Colour Space Transform and get this:

It looks somehow like it's dull, but also too saturated and too contrasty, both at the same time.  and like there's a coloured sheen all over everything.

So, you apply some manual white balance before the CST, and play with a bit of adjustment after the CST to try and liven it up a bit:

and here we have the guys skin tones looking way too saturated (he looked perfectly normal in real-life), and it still looks drab and too saturated at the same time.

So, you abandon the CST wondering if there's something strange in it and go full-manual, applying WB, contrast, and saturation.

Which gets you back to roughly the same place.  Its an image of contradiction.  His shirt and jeans are dull, but his skin colours are too saturated.  The wooden bench in the background looks fine (which is telling - wood is very sensitive to grading), but the bike and the flowers in the ladies dress are practically radioactive.  The blue on the bike might be a colour science trick to emphasise blue skies and maybe isn't meant to be realistic, plus we can scale it back with a Hue vs Sat or a local adjustment, but the flowers on the dress are practically skin coloured, so they can't be a deliberate special effect built into the colour science, lest everyone from earth seem like plasticine lifeforms.

So, we look deeper, trying to understand what's going on.  I've seen images like this since, and it normally means you're stuffing up the WB.  The ladies shoes seem like a good reference point, looking like they were made to be white and also that they'd be well cared for and probably a reliable reference point.

So, we use them as a WB point and we get this:

A complete success in the sense that her shoes are now white and more stylish than ever, and a complete failure in the sense that the rest of the image is obviously completely screwed.

If we were paying close attention (and I can do this because I zoomed in very closely to her shoes) we'll note that they weren't all one colour - by making them white the parts of her shoes that were darker and in shadows became very blue, just like the whole rest of the image.  We can also look for other clues, like the cover on the bicycle seat.  Here is a close-up from the previous grade where we adjusted everything manually:

Now we starting to understand what we're dealing with.  For a start, we have horrific macro-blocking.

This is what happens when you don't listen to the advice of your elders when they say that 8-bit Log is a bad idea, and just to throw caution to the wind, you also filmed with a horrifically low-bitrate codec, in slow-motion, in mixed lighting conditions, with your subject in deep shadows.  If I was a professional film-maker I would definitely be going to hell for this.  As an amateur I might still have an uphill battle talking myself out of eternal damnation!

So, placing my fate on the back-burner we turn back to the bicycle seat cover, if we assume that the green and blue/purple tint here aren't complete fiction (they're not as they're in all frames so aren't random noise in just this frame) then it shows we have some strange things going on.  

We have found strange green/purple issues going on, and also an un-explained warm-tint as well.  At this point, I should confess I've been hiding something.  This is another shot at the same location, showing the building that the violinist is facing, and we find the cause of why the light might be a little 'warm':

Is that film-maker sin number 5?  I should stop counting.

So, we have to accept that the scene is un-naturally warm, but we still need to turn out attention to the green/purple issues.  By playing with the colours in the Shadows and Highlights wheels we can clean it up a little:

It's better, but still just, wrong.  Look at the colours in the windows of the building top-right of frame.  Look at the CA on the satellite dish.  Look at the macro-blocking on the skin of the violinist or the lady sitting.

While trying to grade this and I learned not to use the 1080 modes, especially the 50p, at least for how I shoot (difficult available lighting and in log profile).

I did some testing, applying the Optical Flow to slow the 4K25p down to 50% speed (as if I'd filmed 50p and conformed to 25p) vs the native 50p and decided that the Optical Flow created a better result than the 50p.  So, never use the 1080 modes again.

This will do it for part one.  I can only attach so many images, and only embarrass myself so much in a single post!

 

[kye]

What about a 4K shot then?  The files are only 8-bit, but the XC10 scored a lot of press from its 305Mpbs 4K.

Here's a 4K shot in (hopefully) much better lighting.  Its of a wedding procession that randomly came through a square in the middle of Florence, complete with full band in front and people throwing confetti etc.

The C-Log:

One of the reasons to shoot 4K is to be able to reframe in post, and considering that 4K is basically 4 1080p frames, even if I zoomed to 200% I should still be looking at a 1080p frame with 76Mbps, which is much better than the 35Mbps(?) of the 1080p25.

So, let's apply a WB, CST, and some contrast and saturation, to get this:

Despite using luminance mapping and saturation mapping on the CST, we have colours that the 80s would be proud of.  Getting a bit smarter we can qualify the more saturated areas and pull them back a bit.  There is also some noise in the brides arm and the grooms jacket, so we can apply some temporal NR (another free feature in Resolve).  

We get this:

Not bad I guess, although more work on the extreme colours would probably pay off.

[kye]

Another shot from a different trip.  C-Log:

With WB and CST applied:

The colours just look strange to me.  The trees on the far bank don't actually look like that, it looks like they're too yellow and the stairs also looks kind of artificially yellow, like I need to push the WB cooler, but looking at the temperature of the people in the shade they wouldn't want to go any cooler or risk looking very strange.  The guys hat bottom left is very blue and even into the shadows.

If I clean that up it looks better, but still odd.  It's only when I put a node before everything else and reduce the contrast in it that things start to look semi-natural:

but, now we're at this faded kind of look again, and it's not back to actually looking normal, it just looks less bad.

I'm beginning to wonder if this is a mixed lighting issue or what it is.  If it's mixed lighting that is caused by shooting where things are in shadow and in sun then that's pretty screwed up, as good luck shooting anything outdoors.

Time for some tests.

[Geoff CB]

Your getting in the ballpark. I think your next step is making sure you watch you scopes / parade and adjust just highlights and shadows using the LOG color wheels.

 

 

[kye]

Now I'm the proud owner of a colour checker, I can put some science into this madness, so today I filmed some test shots, with a handy tree in the shot to see what makes it radioactive and what doesn't.

C-Log 4K 305Mbps shot:

If we manually WB on the white square and apply a straight CST without luma or sat mapping we get this postcard from our nuclear fallout future:

If we dial back the saturation after the CST to get something in the ballpark of this world, we get this:

Not a bad image, and the tree looks pretty close to how it actually looks, although I'm tempted to say it's a bit blue.

If we instead of all this we get Resolve to look at the colour checker and auto-magic us into perfection, we get this:

Potentially a more lifelike rendition, but more due to the exposure rather than the colours.

We get this if we abandon all forms of magic tools and just do it ourselves by setting WB, white and black points, then gamma, then a realistic amount of saturation:

This potentially looks the most lifelike.

[kye]

What about WB being wrong in camera and corrected in post?

Custom WB from colour-checker with manual processing from above:

Auto WB from XC10 with WB corrected in post (identical processing):

Deliberately wrong WB from XC10 (I selected "overcast") with WB corrected in post (identical processing):

We learn two things from this.  The first is that it didn't choose the right WB, and the second is that WB in-camera matters as the second and third images both have cool-tinted shadows (look at the neighbours fence bottom left - it's neutral in the custom WB shot but not in the others).

[Geoff CB]

2 minutes ago, kye said:

Now I'm the proud owner of a colour checker, I can put some science into this madness, so today I filmed some test shots, with a handy tree in the shot to see what makes it radioactive and what doesn't.

C-Log 4K 305Mbps shot:

If we manually WB on the white square and apply a straight CST without luma or sat mapping we get this postcard from our nuclear fallout future:

 

If we dial back the saturation after the CST to get something in the ballpark of this world, we get this:

Not a bad image, and the tree looks pretty close to how it actually looks, although I'm tempted to say it's a bit blue.

If we instead of all this we get Resolve to look at the colour checker and auto-magic us into perfection, we get this:

 

Couple notes:
That shot is very underexposed, and you want the bottom right of the color chart to be the brightest part of your entire frame. It should NOT be in shadow.

[kye]

1 hour ago, kye said:

 

 

Mine above, for comparison.

19 minutes ago, Geoff CB said:

Your getting in the ballpark. I think your next step is making sure you watch you scopes / parade and adjust just highlights and shadows using the LOG color wheels.

Thanks, yours is definitely better than mine.  The guys hat in yours isn't nearly so artificially saturated, but in some ways yours is more saturated than mine, or at least equal.

Interesting that your method, at least from your one-line description, is similar to the manual method on the test shots.

[kye]

5 minutes ago, Geoff CB said:

Couple notes:
That shot is very underexposed, and you want the bottom right of the color chart to be the brightest part of your entire frame. It should NOT be in shadow.

Ah!  So it should have been angled so it was in full-sun?  That's useful to know

The highlight on my forehead was by far the brightest point of the image, as I took it.  Considering it was a specular highlight it's likely to beat the chart, but clipping things like that is probably ok.

Possibly most importantly is that that's all auto-exposure on the part of the XC10.  Which is how I used to shoot with it, and how most of my footage looks.

On an unrelated note, it's very strange taking the obviously over-saturated images from the CST and desaturating them, the only thing that seems to move is the colours in the colour chart, and almost everything else stays basically the same.

[kye]

In contrast, here's an equivalent shot from the GH5 with HLG and a basic WB, black and white point correction, and saturation adjustment:

Obviously the gamma curve is different, and sat is a bit higher, but interesting comparison.

I was looking at the vector plots of each, but nothing immediately useful stood out.

[Geoff CB]

14 minutes ago, kye said:

Mine above, for comparison.

Thanks, yours is definitely better than mine.  The guys hat in yours isn't nearly so artificially saturated, but in some ways yours is more saturated than mine, or at least equal.

Interesting that your method, at least from your one-line description, is similar to the manual method on the test shots.

So your scopes have the blacks a little above actual black. You can see the bottom of your green and blue channels are above your red channel, it's why your shadows appear slightly blue.

 

This is my adjustment, bringing down blacks slightly and warming them up.

[thebrothersthre3]

I am thinking the funky colors in the violin shot are due to underexposure. It looks like a rather high dynamic range scene. To get good exposure on your subject you'd probably have to clip the sky in the back. Underexposure can result in funky colors. 

[tupp]

Generally, the darker the image, the more the saturation.  Furthermore, most digital cameras give a lot of saturation in their non-raw files, and Canon cameras additionally boost the reds.

 

Starting with the brightest sample posted, the image below was yielded merely by boosting the gamma/mid-tones, bringing the blacks down to zero, reducing the saturation and backing off the reds (for Canon):

 

 

If one wants to keep it a little darker (and still have it look like daytime), be more gentle in boosting the mid-tones but further reduce the saturation, and keep the blacks at zero and keep the Canon reds reduced as in the image directly above:

 

 

By the way, the fringing/chromatic-aberration doesn't look too bad, and a light touch with a CA/fringe filter should take care of it nicely.

[Mark Romero 2]

I am not familiar with Canon LOG, but I know if I were shooting the same shots you took in italy with either my a6500 or my Panasonic S1, they would have been significantly underexposed and a lot tougher to grade. (I know, I know, what's done is done, but since it seems like a nice little camera, maybe worth trying out some things to see whether you can use it in the future, too).

Also, if you are doing a CST in a node in Resolve (as opposed to doing it at the project settings level), then I find that doing my exposure adjustments in the first node using the LOG controls / contrast / pivot / etc, then have the CST as a second node, then do the actual color balance and grading in a third node can be helpful. I could be wrong, but my understanding is that the CST transform is applied as the LAST effect in any single node. So (again, from what I understand) if you are changing  colors and have a CST in the same node, the color is being changed while still in LOG, not in RGB.)

I then add a fourth node set to LAB cie color space, disable channels 2 and 3, and do my sharpening in that node to help avoid excessive chroma noise when sharpening. (To clarify, that fourth node is not a CST from RGB into LAB, it is just a node where the Color Space is set to LAB CIE)

And finally, the last node I usually have is in RGB but I usually turn off two of the three channels and just use it to target saturation and Luminence of one particular channel. Might be good to do any masking / keying in this node as well... but need to do more testing to be certain.

It ain't gospel, mind you, but it seems like a fairly good working arrangement for me.

[tupp]

1 hour ago, tupp said:

Generally, the darker the image, the more the saturation.

Meant to say, "The darker the image, the more saturation seems conspicuous -- unless, of course, the image gets so dark that it's mostly black."

[BenEricson]

5 hours ago, thebrothersthre3 said:

I am thinking the funky colors in the violin shot are due to underexposure. It looks like a rather high dynamic range scene. To get good exposure on your subject you'd probably have to clip the sky in the back. Underexposure can result in funky colors. 

Exactly. The exposure needs to be motivated. It is difficult to force shots that don't exist, especially if you're trying to match contrast through out an entire piece. This is why shooting log footage without a LUT gets confusing.

Different colors will appear differently depending on how they are exposed, obviously. The Canon, Red, and Arri all make sure green hold saturation even at the top of the image. All the LUTS you will use are probably balanced for a more standard exposure. If you're doing things like pushing highlights and extreme ETTR, you'll run into problems with saturation on certain colors.

8 hours ago, kye said:

Now I'm the proud owner of a colour checker, I can put some science into this madness, so today I filmed some test shots, with a handy tree in the shot to see what makes it radioactive and what doesn't.

C-Log 4K 305Mbps shot:

Looking at your frames makes me want one of these cameras again. I really love how the footage looks. I owned one 3 years ago. I remember I messed with some of the baked in profiles, those looked really nice as well. Maybe Production Standard? I can't remember. I use the C300 Mk2 now so I get them mixed up. The color seemed a bit thicker than if I shot C-LOG and tried to match what a baked in profile was giving me.

You can do a lot in post with those files but you need to know what you want to do when you get into the edit. With the violinist,  I would assume you'd want to compensate for the under exposure and then maybe throw some sort of "power windows" or gradient exposure on the clouds to keep some of that beautiful blue sky and green trees.

With the portrait test shot / you're essentially exposing for the shadows because of how the light is hitting you. If you opened up to your left slightly and had a bit of diffusion, you could drop the whole image down by a few stops and get a more rich / darker background. As is, the light is kind of a back light / side light.

Here is what I came up with on the two frames. I don't have resolve, this is just Magic Bullet Looks in Premiere Pro. I don't have a color checker, but your WB is great so it's not that difficult to just level it out and tweak a few color with the HSL sliders. I did my best to try to match the two frame just for fun.

 

[kaylee]

hi kye!

my 2 cents is only that it seems very... Canon 

i LIKe that, as far as getting an image that im happy grading, but it can be an oversaturated wonky mess to begin with. my brown dog turns orange, etc. in the future i bet there will be a gimmick where you can dial in saturation to taste for faces, landscapes, etc.  i know thats a tall order but hey why not 😂

that guys magenta shirt is a perfect example, u kno?

smart folks have pointed out that shadow areas should prolly be desaturated in general, id say even more so with canon. i look at it as the nature of the beast, im sure with the eos r5/6 i would be doing a lot of power windows 😂

but its all to taste. speaking of which

On 6/21/2020 at 2:51 AM, kye said:

grab a drink

i did. im having a michelada. i kinda like it! its like a mexican beer bloody mary... p good! not enough booze tho,  ill add a shot next time

[austinchimp]

My first impression of your travel video is that you uploaded it in an extremely low bitrate, which does it a big disservice. Could you upload it at 20,000 - 40-000 kbps? I downloaded the ungraded still of the man playing the violin and it also looks like 720p, which again doesn't help.

To me, the color space transformed shot looks about right for the lighting, the violinist is obviously very much in the shade, which you've exposed to keep the highlights. I think there's just too much dynamic range in this shot for the camera to expose the skin tones nicely, as they're firmly in the shadows.

The good news is that I don't think it looks terrible! If those were the lighting conditions then that's what it was, and the way you exposed it avoids any nasty highlight clipping. If the man was standing in the shade, then it's ok for him to look like he's in the shade. He doesn't have to look like he has a fill light.

Maybe try playing with the curves a little to bring up the skin tones if possible, while keeping some blacks. 

[kye]

Driving home yesterday I saw a woman out walking with a bright orange coat and I realised that the scene before me was pretty much how the camera rendered the wedding shot in Italy - everything very low in saturation except a few spots of radical colour that basically don't look real.  I guess that's how the real world actually looks.

It's not how the world looks in most videos, and it's definitely not how we experience the world.

On 6/21/2020 at 8:57 PM, Geoff CB said:

So your scopes have the blacks a little above actual black. You can see the bottom of your green and blue channels are above your red channel, it's why your shadows appear slightly blue.

I've had issues before with trying to get my blacks to sit at the right level.  It seems like the difference between crushed and instagram-retro is less than the thickness of a human hair.  Maybe I need to apply a node that boosts the signal by 10x then go through my whole timeline to equalise everything out and then remove the 10X node again.  

I sometimes do that with saturation, having a node that radically saturates the whole timeline and then I can look at all the thumbnails and see any clips that have an obviously different WB, which I can then fix and remove the node again.

On 6/21/2020 at 10:50 PM, thebrothersthre3 said:

I am thinking the funky colors in the violin shot are due to underexposure. It looks like a rather high dynamic range scene. To get good exposure on your subject you'd probably have to clip the sky in the back. Underexposure can result in funky colors. 

I'm committing every sin in the violinist shot.  Underexposing a mixed-lighting scene with an 8-bit camera in log using a codec with a low bitrate.....  in slow-motion.  This specific shot taught me the value of higher bitrates and 10-bit colour.

Unfortunately, this is what real-life actually looks like, but the problem is that it's not how we perceive the real-world, and due to the technical limitations involved I can't get this shot to even remotely how it looked to me at the time.

On 6/22/2020 at 2:21 AM, tupp said:

Starting with the brightest sample posted, the image below was yielded merely by boosting the gamma/mid-tones, bringing the blacks down to zero, reducing the saturation and backing off the reds (for Canon):

 

Yes, that looks more neutral.

The issue is that this isn't remotely how the scene looked in real life.  

My frustration is that I can completely remove all colour issues by simply reducing the saturation to zero, but the target aesthetic of this piece was that it was a happy occasion and Italy is a colourful place.  The grade above simply doesn't match that aesthetic, being closer to a bleach bypass look for a desolate film rather than a colourful look for a happy rom-com.

For context, this trip to Italy was part of my honeymoon!

On 6/22/2020 at 3:38 AM, Mark Romero 2 said:

I am not familiar with Canon LOG, but I know if I were shooting the same shots you took in italy with either my a6500 or my Panasonic S1, they would have been significantly underexposed and a lot tougher to grade. (I know, I know, what's done is done, but since it seems like a nice little camera, maybe worth trying out some things to see whether you can use it in the future, too).

Thanks for outlining your process, I'll have to replicate it and have a play.

You think that the S1 shot would have been harder to grade?  That surprises me, considering that my GH5 was a breath of fresh air after trying to grade the XC10 footage for a couple of years.  

My experience of watching colour grading videos online with footage from five-figure cine cameras was that you can raise the levels up and up and up and they just scale up but otherwise look completely fine, rather than pulling things up a bit and being presented with awful hues and terrible artefacts.  The GH5 feels exactly the same way in post as that - it just scales up without any drama.  I can understand the a6500 not doing so well though.

On 6/22/2020 at 4:14 AM, BenEricson said:

Exactly. The exposure needs to be motivated. It is difficult to force shots that don't exist, especially if you're trying to match contrast through out an entire piece. This is why shooting log footage without a LUT gets confusing.

I guess I don't want to force a shot that doesn't exist, but it did exist, the camera just failed to capture it even remotely the way that it was, in reality.

I've found the GH5 to be far more forgiving of difficult lighting.  Mind you, had I used the XC10 4K mode then it might have been a different story due to the higher resolution and bitrate of three times more bits per pixel, or 12 times more bits per frame.

23 hours ago, kaylee said:

hi kye!

my 2 cents is only that it seems very... Canon 

i LIKe that, as far as getting an image that im happy grading, but it can be an oversaturated wonky mess to begin with. my brown dog turns orange, etc. in the future i bet there will be a gimmick where you can dial in saturation to taste for faces, landscapes, etc.  i know thats a tall order but hey why not 😂

that guys magenta shirt is a perfect example, u kno?

smart folks have pointed out that shadow areas should prolly be desaturated in general, id say even more so with canon. i look at it as the nature of the beast, im sure with the eos r5/6 i would be doing a lot of power windows 😂

but its all to taste. speaking of which

i did. im having a michelada. i kinda like it! its like a mexican beer bloody mary... p good! not enough booze tho,  ill add a shot next time

Yeah, very Canon.  I really do make life difficult for myself by shooting in the worst possible conditions!

lol, I immediately thought of vodka, but perhaps you should be adding tequila?

11 hours ago, austinchimp said:

My first impression of your travel video is that you uploaded it in an extremely low bitrate, which does it a big disservice. Could you upload it at 20,000 - 40-000 kbps? I downloaded the ungraded still of the man playing the violin and it also looks like 720p, which again doesn't help.

To me, the color space transformed shot looks about right for the lighting, the violinist is obviously very much in the shade, which you've exposed to keep the highlights. I think there's just too much dynamic range in this shot for the camera to expose the skin tones nicely, as they're firmly in the shadows.

The good news is that I don't think it looks terrible! If those were the lighting conditions then that's what it was, and the way you exposed it avoids any nasty highlight clipping. If the man was standing in the shade, then it's ok for him to look like he's in the shade. He doesn't have to look like he has a fill light.

Maybe try playing with the curves a little to bring up the skin tones if possible, while keeping some blacks. 

Not too sure what the YT upload was at, but the frames I posted above were 720 to be a bit easier to view in the thread.

Here's the CLog still at the original resolution:

The 1080p is pretty soft, and the 1080p50 even more-so.  I exported the PNG above and a TIFF file and swapping back and forth between them shows very little difference.

For reference, here's another CLog shot from the video, PNG at 4K:

[Mark Romero 2]

5 hours ago, kye said:

You think that the S1 shot would have been harder to grade?

What I meant to say is that significantly underexposed shots (regardless of camera) are harder to grade, and I seem to get more saturation and the colors tend to shift a bit.