Big blind test - what should I do?

[kye]

I've been working on a project to match the GH5 to the BMMCC for some time now, and I've decided the next step is for a blind test.

In this blind test I will show a series of clips shot with either the GH5 or BMMCC, using one of the many lenses I have in my collection, where the BMMCC will have a basic colour grade applied and the GH5 will be graded to have similar colour to the BMMCC.  There will be an anonymous poll where people can guess if they think that it was the GH5 or the BMMCC.

Due to various technicalities, I have 20 camera/lens combinations, and I have been trying to shoot each combination at a range of apertures as lenses are often softer or sharper based on aperture, and sharpness is one of the things I'm matching between the GH5 and BMMCC, so lenses play a role in this.

If I was to shoot each camera/lens combination with aperture wide-open, with aperture stop down, have a shot with some clipping and likely flare, and a shot with some skin-tones in it, that would mean I may have as many as many as 80 shots, which is a lot.  It wouldn't be practical to ask 80 questions in a poll, so I'm thinking I'll group multiple shots together into batches.

Also, I'm not shooting identical scenes side-by-side, because that's not what matters.  If we make a film and someone watches it and can't tell if you used an Alexa or not then it's job well done.  The fact they might be able to tell between your camera and an Alexa in a side-by-side test is irrelevant when they're watching your film.
Therefore, I've been shooting test shots over a period of days in varying lighting situations.

So the test would be a "batches" of shots, where a batch would be shot with one camera, but would include a mixture of lenses, apertures, subjects and lighting situations.  I would then have the quiz question....   

What camera do you think Batch X was shot on: 

• Definitely the BMMCC
• Maybe the BMMCC but I'm not sure
• No clue!
• Maybe the GH5 but I'm not sure
• Definitely the GH5

If I had 80 shots grouped into batches of 8 shots, then that would be 10 batches, which is far more manageable.

So, here's my question and the purpose of this post.

While I'm doing this, what other questions should I be asking?  It's a blind test between two cameras, but it's also a blind test with many lenses that probably aren't directly compared.  Obviously I'll create a thread and we can talk about it for a couple of weeks before I reveal the results, but the comments in the thread aren't anonymous and I think people will hold back in fear of saying they like the cheapest lens vs the most expensive or trendiest lens, so some anonymous quiz questions might also be useful.

Should I ask things like:

• Which shots are your favourites?  (text entry so people can list as many as they like)
• What shots are noteworthy and what do you think about them?    (text entry so people can say whatever they like)

Or maybe I should number the lenses in the test footage (eg, lens #4) so that I can ask people which lens was their favourite?

For every shot I'm making a note of which camera, what lens, and what aperture it was set at, and what focal length was set (if it's a zoom).  We can also tell from the shot if it's close focus or more distant.  It won't be a complete dataset as doing every combination would be thousands of shots, but it might be interesting anyway.

What do you think?

[BTM_Pix]

On 8/8/2020 at 4:59 AM, kye said:

What do you think?

I'm thinking about the sort of test that you might consider to satisfy this objective that you described in the post.

"If we make a film and someone watches it and can't tell if you used an Alexa or not then it's job well done.  The fact they might be able to tell between your camera and an Alexa in a side-by-side test is irrelevant when they're watching your film."

I honestly think that the only two options that are required as responses in that context would be :

1. I don't care what it was shot on, I like it.
2. I don't care what it was shot on, I don't like it.

You might possibly learn more about the cameras themselves with the greater granularity of answers but you'll learn far more about the reaction to what you are doing with them with only those two options.

There's nothing to say you can't do both but I would give serious consideration to doing this simpler subjective test first and then use the more elaborate optioned answer version to make it more objective for whatever it is you want to find out about the cameras.