Blind tests - Cameras, lenses, resolutions, codecs, crop factors, etc...

[kye]

There are a number of blind camera tests around the place, and I find them useful to compare your image preferences (instead of the prejudices we all have!) so thought collecting them in a single thread might be useful.

To get philosophical for a second, I think that educating your eye is of paramount importance.  It's easy to "train" your eye through the endless cycle of 1) hear a new camera is released, 2) read the specs and hear about the price 3) build up a bunch of preconceived notions about how good the image will be, 4) see test footage, 5) mentally assume that the images you saw must fit with the positive impression that you created based solely on the specs and price, and 6) repeat for every new camera that is released.  That's a great way to train yourself to think that over-sharpened rubbish looks "best".

The alternative to this is evaluating images based solely on the image, and to go by feel, rather than pixel-peeping based on spec/price.  To this end, it's useful to view blind tests of cameras you can't afford, lenses you can't afford, and old cameras you turn your nose up at because they're not the latest specs.  News flash, the Alexa Classic doesn't have the latest specs either, so how many cameras have you dismissed based on specs but were making an exception for the Alexa all this time?

Also, although you may find that you like a particular cine lens but can't afford it (or basically any cine lens for that matter), often the cine lenses have the same glass as lenses that cost a tenth or less of the cine version.  Furthermore, you may be able to triangulate that you like lenses or a camera / codec combo that gives a particular image look, and perhaps that look can be created by lighting differently, or using filters, or changing the focal lengths you use.  Educating your eye can literally lead to you getting better images from what you have without purchasing anything.

To kick things off, here's Tom Antos' most recent test, with Sony FX3, Sony FX6 BM Pocket Cinema Camera 6K Pro, RED Komodo, and Z-Cam E2 F6

Test footage:

and the results and discussion:

Here are his tests from 2019 - BM Pocket 6K, Arri Alexa, RED Raven, Ursa Mini Pro

Test footage:

and results and discussion:

Another blind test from TECH Rehab, comparing Sony F65, Sony F55, Arri Alexa, Kinefinity Mavo 6K, BM Ursa 4K, BMPCC 6K, BMPCC 4K

Test footage:

and results:

Another test from Carls Cinema, comparing OG BMPCC 2K and BMPCC 4K:

Another test from Carls Cinema comparing OG BMPCC 2K to GH5:

A big shootout from @Mattias Burling comparing a bunch of cameras, but interestingly, also comparing different modes / resolutions of the cameras, and also paired with different lenses because (hold the front page!) the camera isn't the only thing that creates the image.  Shocking I know....  

I won't name the cameras here, as not even knowing which cameras are in there is part of the test.

The test footage:

And the results and discussion:

More camera tests:

Another great camera/lens combo test, this time from @John Brawley:

 

And a blind lens test:

If anyone can find the large blind test from 2014 (IIRC?) that included the GH4 as well as a bunch of cine cameras, it would be great to link to it here.  I searched for it but all I could find was a few articles that included private vimeo videos, so maybe it's been taken down?  It was a very interesting test and definitely worth including.

If you know of more, please share! 🙂

[noone]

20 hours ago, kye said:

 

 

If anyone can find the large blind test from 2014 (IIRC?) that included the GH4 as well as a bunch of cine cameras, it would be great to link to it here.  I searched for it but all I could find was a few articles that included private vimeo videos, so maybe it's been taken down?  It was a very interesting test and definitely worth including.

If you know of more, please share! 🙂

Two stand out lenses for me in that test (the 40mm Baush and Lomb and the Cooke ....then i looked at the prices... OUCH!

[kye]

4 hours ago, noone said:

Two stand out lenses for me in that test (the 40mm Baush and Lomb and the Cooke ....then i looked at the prices... OUCH!

Awesome...  next steps are:

1. figure out of there is a cheaper way to get that glass, perhaps in a consumer lens
2. figure out if there is a different way to get the same optical recipe (for example the soviet lenses are famously replications of the Zeiss recipes that the soviets took from Germany at the end of WWII)
3. figure out what the image qualities are that you like from those lenses and work out if there is a way to replicate them in other ways, like diffusion filters, streak filters, etc

[PannySVHS]

8 hours ago, kye said:

Awesome...  next steps are:

1. figure out of there is a cheaper way to get that glass, perhaps in a consumer lens
2. figure out if there is a different way to get the same optical recipe (for example the soviet lenses are famously replications of the Zeiss recipes that the soviets took from Germany at the end of WWII)
3. figure out what the image qualities are that you like from those lenses and work out if there is a way to replicate them in other ways, like diffusion filters, streak filters, etc

Awesome, but psst.:)

[TomTheDP]

I picked B and D with the Tom Antos test. Which was the FX3 and RED Komodo.

The FX3 and FX6 shouldn't look different color wise, weird. These test are fun although I don't find them that useful. Maybe I should go buy an FX3 or A7S3 now lol. 

[kye]

10 hours ago, TomTheDP said:

I picked B and D with the Tom Antos test. Which was the FX3 and RED Komodo.

The FX3 and FX6 shouldn't look different color wise, weird. These test are fun although I don't find them that useful. Maybe I should go buy an FX3 or A7S3 now lol. 

My preference was A, C, E, D, B.  

The more of these that I watch the more I realise I'm looking at the colour, and in this test I didn't like the green reflection that wasn't there in real life.  Of course, as they were all shot in very neutral codecs this should be editable in post relatively easily prior to the 709 conversion.

But any of these cameras can create a great image, as Tom said.

[noone]

On 4/4/2021 at 2:25 PM, kye said:

Awesome...  next steps are:

1. figure out of there is a cheaper way to get that glass, perhaps in a consumer lens
2. figure out if there is a different way to get the same optical recipe (for example the soviet lenses are famously replications of the Zeiss recipes that the soviets took from Germany at the end of WWII)
3. figure out what the image qualities are that you like from those lenses and work out if there is a way to replicate them in other ways, like diffusion filters, streak filters, etc

Nah.       I watched it out of curiosity.     Those two just looked nice to me in that setting.

I do not need any new lenses and non I really want in that focal range.

I have gotten rid of a lot of gear lately (given away) and have come to realise I could be very happy with just a handful of lenses for my (mostly stills but some video) uses.

I could probably actually do 90% of my shooting with just my 17 TSE and 55 1.8 and 75% with just the TS-E (or at least almost all of my photos used by others are with the 17 TS-E).

I do still want an AF portrait lens of some description (85mm plus).       My little RX100 iv has taken care of a lot of other use now.

[Mark Romero 2]

For the Tom Antos tests, I had my prefered list as pretty similar to what everyone else had (except switch around alexa and Red)

1) Ursa Mini Pro

2) BMPCC 6K

3) Red

4) Arri

Mind you, I was looking primarily at skin tones.

Maybe I am losing my mind, but the Arri skin tones just looked so green in that test.

[TomTheDP]

3 minutes ago, Mark Romero 2 said:

For the Tom Antos tests, I had my prefered list as pretty similar to what everyone else had (except switch around alexa and Red)

1) Ursa Mini Pro

2) BMPCC 6K

3) Red

4) Arri

Mind you, I was looking primarily at skin tones.

Maybe I am losing my mind, but the Arri skin tones just looked so green in that test.

Arri does tend to have a green push. 

[kye]

1 hour ago, Mark Romero 2 said:

Mind you, I was looking primarily at skin tones.

Maybe I am losing my mind, but the Arri skin tones just looked so green in that test.

I'm also just looking at skin tones.  I should probably try to zoom out a little, but it's interesting to see how we each perceive these things.

As @TomTheDP said, Alexas are commonly a bit green.  I was completely surprised when I heard this for the first time because you never see it in the final footage, but it's just a thing that everyone deals with apparently.  

I think that knowledge of colour grading is perhaps the biggest differentiator between how primarily amateur groups and primarily pro groups discuss image quality - the pros seem to view SOOC footage as a raw material whereas amateurs discuss it like it's a final product (or it's only a LUT away from a final product).  
I saw someone on another forum comparing two RAW-shooting camera models from the same company and their comment was that they really wanted to like the later model but there was a slight texture to the skin tones they didn't care for.  The thing is that both the cameras share the same sensor, the person who made the test used the same settings and just applied a technical LUT to get a straight comparison, and they were taken outdoors non-simultaneously so small differences between them were inevitable.  They were talking like the 12-bit RAW footage wasn't changeable, yet it is the most neutral flexible codec available.
Even the way that cinematographers do latitude tests on cameras that shoot RAW indicates they're viewing the camera with a "shoot it so after colouring it I get the best image" but that mindset is almost completely absent elsewhere, other than the odd cult who have sold all their possessions to this deity they don't understand called ETTR.

It's a bizarre world where the leaders are running around screaming "shoot in LOG so your footage looks miserable SOOC, but let's not ever talk about colour grading - you should buy my LUT instead!" and no-one questions this.  You'd imagine a counter-culture would have emerged by now, but unfortunately it seems that if anyone is rejecting this premise they haven't done it by empowering themselves by learning about grading..

[Mark Romero 2]

18 hours ago, TomTheDP said:

Arri does tend to have a green push. 

So it looks like Tom Antos and friends had a revised test, and the cameras favored the most by viewers were... 

The FX3 and FX6

So either the new Sony cameras have sparkling fresh colors, or so much for the myth of color science.

[Mark Romero 2]

16 hours ago, kye said:

I'm also just looking at skin tones.  I should probably try to zoom out a little, but it's interesting to see how we each perceive these things.

As @TomTheDP said, Alexas are commonly a bit green.  I was completely surprised when I heard this for the first time because you never see it in the final footage, but it's just a thing that everyone deals with apparently.  

I think that knowledge of colour grading is perhaps the biggest differentiator between how primarily amateur groups and primarily pro groups discuss image quality - the pros seem to view SOOC footage as a raw material whereas amateurs discuss it like it's a final product (or it's only a LUT away from a final product).  
I saw someone on another forum comparing two RAW-shooting camera models from the same company and their comment was that they really wanted to like the later model but there was a slight texture to the skin tones they didn't care for.  The thing is that both the cameras share the same sensor, the person who made the test used the same settings and just applied a technical LUT to get a straight comparison, and they were taken outdoors non-simultaneously so small differences between them were inevitable.  They were talking like the 12-bit RAW footage wasn't changeable, yet it is the most neutral flexible codec available.
Even the way that cinematographers do latitude tests on cameras that shoot RAW indicates they're viewing the camera with a "shoot it so after colouring it I get the best image" but that mindset is almost completely absent elsewhere, other than the odd cult who have sold all their possessions to this deity they don't understand called ETTR.

It's a bizarre world where the leaders are running around screaming "shoot in LOG so your footage looks miserable SOOC, but let's not ever talk about colour grading - you should buy my LUT instead!" and no-one questions this.  You'd imagine a counter-culture would have emerged by now, but unfortunately it seems that if anyone is rejecting this premise they haven't done it by empowering themselves by learning about grading..

You are right. I don't remember going to see a lot of movies shot on an Alexa and saying, "wow, those skin tones look green," unless there was a scene under fluorescent lighting where things are SUPPOSED to have a green tint.

But your larger point that there is a BIG difference between what you get SOOC and what the footage looks like in the hands of a pro colorist is absolutely spot on. 

I guess these kinds of tests have a point; if you are not going to be doing some serious color grading, then certain cameras are going to be better for your production than others. 

Just don't compare SOOC with what can be done with the footage in the hands of a competent colorist.

[MrSMW]

24 minutes ago, Mark Romero 2 said:

So it looks like Tom Antos and friends had a revised test, and the cameras favored the most by viewers were... 

The FX3 and FX6

So either the new Sony cameras have sparkling fresh colors, or so much for the myth of color science.

When Toneh Northrup did a blind test on all major manufacturers, didn't Sony come out on top there also?

I seem to remember...

[Mark Romero 2]

1 hour ago, MrSMW said:

When Toneh Northrup did a blind test on all major manufacturers, didn't Sony come out on top there also?

I seem to remember...

You are correct, sir.

In another test, another youtuber showed a lot of non-photographers lots of photos, some with a very  shallow depth of field, and some with "normal" depth of field, and overwhelmingly the people preferred the photos with MORE depth of field over the photos with shallow depth of field. 

 

So there we have it.

6K and 8K Resolution??? 

Overrated.

Straight out of camera colors???

Overrated.

Super shallow depth of field (aka, Cinematic Look)???

Overrated.

Oh, and then there is the video by Jordan Drake where he says he has been "living in denial" or something like that because he shot a video and purposely didn't use the 180-degree shutter rule, and no one seemed to notice. (Everyone seemed to notice instead their use of gee-whiz transitions and jump cuts.)

[kye]

4 hours ago, Mark Romero 2 said:

So there we have it.

6K and 8K Resolution??? 

Overrated.

Straight out of camera colors???

Overrated.

Super shallow depth of field (aka, Cinematic Look)???

Overrated.

Oh, and then there is the video by Jordan Drake where he says he has been "living in denial" or something like that because he shot a video and purposely didn't use the 180-degree shutter rule, and no one seemed to notice. (Everyone seemed to notice instead their use of gee-whiz transitions and jump cuts.)

Goodness!  We'll be talking about content next!!  What has the state of the camera forums come to?!?!?!

6 hours ago, Mark Romero 2 said:

Just don't compare SOOC with what can be done with the footage in the hands of a competent colorist.

Just imagine what people could create if they buy cameras that have a thick and luscious image to begin with, AND ALSO learn to colour grade...  

[Mark Romero 2]

8 minutes ago, kye said:

Goodness!  We'll be talking about content next!!  What has the state of the camera forums come to?!?!?!

Just imagine what people could create if they buy cameras that have a thick and luscious image to begin with, AND ALSO learn to colour grade...  

Well... I would do that... but I am too busy saving up for the new Sony 50mm f/1.2 so I can take some epic footage of my cat...

[MrSMW]

11 hours ago, Mark Romero 2 said:

Oh, and then there is the video by Jordan Drake where he says he has been "living in denial" or something like that because he shot a video and purposely didn't use the 180-degree shutter rule, and no one seemed to notice.

I’ve been ignoring it my entire career and no one has ever noticed. Or at least commented.

Without an auto internal ND, it makes true hybrid coverage very difficult...which is what I mostly do.

I’ve never chased the next ‘latest & greatest’ spec. Chased certain standards at times yes and for the last few years, that has been 4K 60P plus great SOOC picture quality (stills & video) that can easily be lightly graded. 

8K? No thanks.

Raw video? Yet to be convinced for my needs.

Super shallow DOF. I do like a shallow DOF and have done my whole 20+ year career, but the main reason I personally prefer faster lenses is more because of the lesser need to crank up the ISO and introduce unwanted noise. Because sensors continually improve in that regard, f2 & f2.8 are now my go to’s whereas just a few years ago, it might have been f1.4 (for full frame).

Great colour SOOC is not negotiable with me however. Anything that requires extensive work to look good isn’t for me.

And in all of the above regards, I’m done camera spec-wise for at least 3.5 years unless something breaks in that time.

As with all these things, it’s just finding that balance of the tools that will best do the job that you need them to do within a certain budget.

Anyway, my tastes must be different to the masses as I took the blind test and Sony didn’t win for me 😜

[Mark Romero 2]

8 hours ago, MrSMW said:

8K? No thanks.

Oh yeah, forgot to mention...

There was some sample 8K video of cherry blossoms and bees posted on youtube the other day.

I tried to watch it on my 2560 X 1440 monitor.

So even after pressing pause, and allowing the video to load halfway first before pressing play again, the video just stuttered along.

My computer isn't that much of a potato: i7-6700, 24GB RAM, RTX 2060 Super with 8GB VRAM, win 10 on a SSD.