iMovie imports GH2 files incorrectly?

[npa201]

I saw the article EOSHD posted the other day but I'm unsure if this is the issue the author was describing.  My current setup is a Macbook Pro 2007 using iMovie '09.  Yes.  I know outdated.  I'm willing to upgrade both software and computer but am trying to wait until this year's Macbook is released (with hopes there will be a retina display). 

Anyway, I've noticed when importing video onto my Macbook that there are significant banding (w/ lots of noise) in darker scenes.  In addition, shots with a lot of bokeh create this issue and I even run into this issue during bright scenes.  I've been trying to isolate what is causing this and I'm a bit unsure what is causing this.  Help would be appreciated in understanding what is happening. 

Below is a video I recently put together of my daughter and the issue pops up several times (times listed below). 

http://vimeo.com/42740933
0:31 - notice the black sweat pants particularly where shadow is cast by the arm
0:59 - the out of focus blacks
1:26 - the blue tile on the top left
1:39 - the most serious offense is my child's leg
2:18 - bottom left - my wife's jeans in the dark area are creating this weird banding/noise issue

Now to be fair, it's a little worse on this Vimeo vid than elsewhere as I've lowered the bit rate considerably to meet the non-pro limitations on uploading.  However, I've noticed these same issues when outputting at 24,000 bits/s (I think that is what a non-hack GH2 is recorded at correct?).  Also, when imported and just playing it while in iMovie '09 i see it.  Unfortunately, I don't have these files in MTS anymore (perhaps I should save those from now on) but when I've shot test footage after noticing these issues and watching it direct from camera to TV and then importing it onto my computer I've noticed these issues are not on the original mts files. 

It appears to me that iMovie is really having an issue understanding large areas where the color is somewhat uniform.  Unfortunately, I love creating that creamy Bokeh (otherwise I would have stuck with a crappy cheap vid camera) and this is where these issues pop up the most.  Any help would be much appreciated. 

Camera gear used in video
Panasonic GH2
Panasonic Leica 25mm f/1.4
Olympus 45mm f/1.8
Fader ND Filter

Computer & Software
Macbook Pro 2007
iMovie '09 (I don't seem to see any option other than import file at original resolution - no other preferences)

[npa201]

One other thing. I haven't messed around with hacking the GH2 to increase the bitrate but was wondering if this could help the situation at all?  Perhaps not since I'm just using iMovie '09? 

[Axel]

Beautiful images with strong, vivid colors and contrasts. Let me first assure you that the banding issue you detected is known by all GH2 users.

[quote author=npa201 link=topic=786.msg5679#msg5679 date=1338093993]Help would be appreciated in understanding what is happening.[/quote] 

I'll try my best. The AVCHD used by the camera is an 8-bit codec. It allowes for 256 steps in luminance between "black" and "white" (actually less, but let's keep it simple). An image that was just recorded with this codec without any intelligent redistribution of values (called "quantization") to where the finest definition is needed, would only show crushed blacks, clipped whites and doughy skin tones. So the processor analyzes all the values and keeps bending the curves - you probably heard about those curves in connection with so-called [i]flat styles[/i]. The quantization is applied to the whole picture. Dark grades ([i]almost[/i] black or [i]almost[/i] uniform) and blurred parts are simplified too much and this causes the banding.

The banding described above is often confused with compression artifacts related to the bitrate. It has nothing to do with it. You will also get banding with a too low rate, but not at 24 mbps (actually lower, but sufficient). I like to emphasize this: The banding won't disappear with a hack!

[quote author=npa201 link=topic=786.msg5679#msg5679 date=1338093993]Unfortunately, I don't have these files in MTS anymore (perhaps I should save those from now on) but when I've shot test footage after noticing these issues and watching it direct from camera to TV and then importing it onto my computer I've noticed these issues are not on the original mts files.[/quote] 

I doubt this, because the phenomenon is typical for the areas and situations where you see the banding. iMovie transcodes AVCHD into the Apple Intermediate Codec (AIC). The original and the copy should be visually indistinguishable. AIC is also 8-bit. When does it cause problems, compared to ProRes (FCP) or original files (Premiere, Vegas, Edius)? Clear answer: When you grade. When you change luminance, hues or saturation. Because iMovie is not able to perform these changes with the proper accuracy, it will [i]then[/i] produce artifacts.

How can you avoid banding in your original files? You must understand that an obviously perfect image is a compromise. If you continue to use iMovie (and you can!), you must try to get the "perfect image" in-camera. It looks as if you have used film-mode settings with high contrast, sharpness and saturation to produce these gorgeous images. This forces the quantizer to rob too many values from the seemingly simple parts. Dial down the values (you have to test it, you should at least never go over zero). For dark scenes use only nostalgic. For these sunny scenes, I would have tried smooth or standard. Try yourself. These are the things you [i]can[/i] influence.

There are more: Avoid too high contrast. [url=http://www.amazon.com/Tiffen-52LC1-52mm-Contrast-Filter/dp/B001U3ZVLE/ref=sr_1_6?ie=UTF8&qid=1338100222&sr=8-6][color=blue]This[/color][/url] filter reduces contrast without influencing the sharpness at all (Tiffen won an Oscar for it). Of course there is a problem with stacking filters, especially with a fader-ND (to avoid reflexes then, you would need a mattebox, the only practical excuse for such a device). It's not easy. But if it where, where was the fun?

> A pro DP would have had the shadows bounced with a reflector to lighten them just a tiny bit, and the banding would disappear, problem solved.
> The bokeh-banding can probably be avoided by another film mode or film mode setting.

One last thing: Any layman who is not obsessed with video technique won't notice the banding. You set yourself high standards for an amateur.

[npa201]

Axel:

Thank you for the response in layman's terms.  What is interesting is when I initially set up my camera I set up all video record to Smooth with I think Sharpness, and Saturation to -2 and Noise Reduction to -2 as well.  I guess I just shot these when it was just really beautiful out and it caused these issues. 

I will try nostalgia for indoor shots as this is where it really bugs me. 
And I will also try out the filter as well. 

Thanks for the info.
Do you think that using any other software like Final Cut Pro (or express) would help improve the output?  Are there good grading tools in it (iMovie really lacks here).
Also, when taking photography I've been advised to underexpose and then pull shadows later on.  Would you suggest the opposite with shooting video and slightly overexpose and just deal with blown highlights?

I appreciate the time and effort in explaining these issues to me.  For the most part, I think my friends/family don't notice the issues but my daughter's legs in the high chair seem so obvious that anyone would notice.

[Axel]

[quote author=npa201 link=topic=786.msg5688#msg5688 date=1338130052]What is interesting is when I initially set up my camera I set up all video record to Smooth with I think Sharpness, and Saturation to -2 and Noise Reduction to -2 as well.  I guess I just shot these when it was just really beautiful out and it caused these issues.[/quote]

Hey, you did it right the first time! With high values the banding would have been [i]worse[/i] then. Solace: With more experience, banding will become very rare!

[quote author=npa201 link=topic=786.msg5688#msg5688 date=1338130052]And I will also try out the filter as well.[/quote]

I have it, but I seldom use it. Why? Because indoors, the contrast is seldom a problem. And outdoors I rather have the ND on and either film
> completely in the shadow
or
> completely in the sun, where I try to capture not too many shadows.
 
[quote author=npa201 link=topic=786.msg5688#msg5688 date=1338130052]Do you think that using any other software like Final Cut Pro (or express) would help improve the output?  Are there good grading tools in it (iMovie really lacks here).[/quote]

FCE (no longer available) used the AIC as well, no difference. FCP Studio used ProRes and had 32-bit floating point color rendering in [i]Color[/i]. FCP X also (inexpensive, but you need a lot of RAM to let it run stable). Improve? If you want to start color grading in earnest, yes. But no software erases banding.

[quote author=npa201 link=topic=786.msg5688#msg5688 date=1338130052]Also, when taking photography I've been advised to underexpose and then pull shadows later on.  Would you suggest the opposite with shooting video and slightly overexpose and just deal with blown highlights?[/quote]

No, you should [i]never[/i] overexpose. You can underexpose *slightly* by watching the histogram in the display/EVF. On the right side, no value may be cut off!

Later on, in iMovie, you may find the clips a little too dark. You can adjust the video by very subtly rising the exposure value (not the brightness). You can grip the right and the left handle of the histogram there and adjust whites and blacks. What is the difference to the bigger applications? iMovie calculates with the 256 integer numbers, the others use the places after the decimal point and only round the results at the very end. If you perform ONE operation in iMovie, the very end is reached, what you see is what you get, and there will be no difference. If you change a second value, the calculation is made on top of the first change, cutting off [i]a lot[/i] of original data already and, after even more changes, possibly resulting in banding and artifacts.

It seems you have a good eye. Your exposure looks perfect. If you always work like this, you could very well save the money for a new software.

[see ya]

Hi, along with camera settings and codec as Axel detailed, banding can be induced at playback on a PC / mac or other device when a conversion to RGB is done, ie: computer monitor.

You mention you don't see the banding going from camera to TV, that would make sense.

The banding may not even be in original MTS files as you mention but such things as:

Handling by decompressing codec, what version of QT are you using assuming iMovie uses QT.

Conversion to intermediate in NLE with the defacto Quicktime levels wrangling either clipping and crushing with older implementations of QT or squeezing with current ones.

8bit precision color processing with assume no RGB parade, scopes or histogram to keep a check on color and luma levels in iMovie. Also assume non color calibrated monitor and no color management available.

Wrong conversion to RGB for preview in the NLE due to graphics card settings ie: BT601/BT709 color matrix skewing appearance of color and contrast which mean we grade judging via a skewed interpretation, not what we wrote to the card in the camera

Rendering to a codec with either no control over final levels at output or a requantized output using a different range of levels to that initially encoded with in camera.

All lead to an output more prone to banding in RGB playback.

A good starting point to find out what iMovie is doing to help you around the situation would be to do a test capture with your preferred camera settings, import it into iMovie and render out to the 'best' codec choice iMovie gives you and compare levels between the two.

Personally I use Avisynth for this as it gives full control over the process without need to second guess what the NLE might be doing, its MS Windows based however.

It is possible to 'improve' any videos you have suffering banding, where there are no MTS files any longer to redo the project by using a debanding filter and gradient rebuilding dithering tool, again Avisynth based plugins with noise reduction -> debanding -> gradient rebuilding at 16bit -> Adding encoder friendly dithering back to 8bit before reencoding.

[Axel]

@yellow
You really seem to know what you are talking about. But once you know typical situations when banding occurs, you can see it in the viewfinder [i]during recording[/i]. There is a fail-safe way to produce banding: Film a fluorescent light on a wall (best at night, when this is the only light), the falloff, the gradient, move the camera up and down. So there [i]is[/i] this problem, and it's not (only) caused by QT, the GPU or a badly calibrated monitor. If you detect during filming, you can change the lighting or the framing and avoid it.

[quote author=yellow link=topic=786.msg5696#msg5696 date=1338151253]It is possible to 'improve' any videos you have suffering banding, where there are no MTS files any longer to redo the project by using a debanding filter and gradient rebuilding dithering tool, again Avisynth based plugins with noise reduction -> debanding -> gradient rebuilding at 16bit -> Adding encoder friendly dithering back to 8bit before reencoding.
[/quote]

Sounds very good. I didn't know about the possibility to "deband" in post. So there is another reason to use Bootcamp, not just the Ptool. Do you recommend Avanti as GUI?

[see ya]

[quote author=Axel link=topic=786.msg5698#msg5698 date=1338187368]
So there [i]is[/i] this problem, and it's not (only) caused by QT, the GPU or a badly calibrated monitor. If you detect during filming, you can change the lighting or the framing and avoid it.[/quote]

Axel, I was not for one minute suggesting that the cause of banding seen was due to the above and not the camera. You already said it has a well known problem, I'm no GH2 owner so I'd already deferred to your more experienced opinion.

The reason for posting was to suggest that there are additional factors that may or may not play a part in banding at playback. Accentuating it, inducing it, 'it' being a 'weakness' in the source video waiting for a bit of 'mishandling' by the NLE to make things worse.

[quote]Sounds very good. I didn't know about the possibility to "deband" in post. So there is another reason to use Bootcamp, not just the Ptool. Do you recommend Avanti as GUI?[/quote]

I use AVSPmod [url=http://forum.doom9.org/showthread.php?t=153248]http://forum.doom9.org/showthread.php?t=153248[/url] for a GUI and used to use Virtualdub.

With this tool, http://forum.doom9.org/showthread.php?p=1386559#post1386559

and/or this http://forum.doom9.org/showthread.php?t=154971 and this http://forum.doom9.org/showthread.php?p=1559222

sometimes using GrainFactory3 16bit mod [15th March in this thread http://blog.niiyan.net/post/19344965089/avisynth-news-on-mar-15-2012] for 'intelligent' grain addition instead of noise.

[Axel]

Thank you!