Best way to shoot a lens comparison?

[kye]

My last lenses are due to arrive, and before I shoot 10 million test images I want to make sure the test is good.

I will be shooting lenses with a wide range of different focal lengths, maximum apertures, and minimum focal distances.  I've done a test mini-shootout that looked like this:

~50mm on SB F2.8:

~40mm no SB F2.8:

~20mm no SB F2.8:

I tried to keep the same framing so had to move the camera back with the longer focal-lengths.  Unfortunately, because of this the DOF is different on all lenses even though those test shots are all F2.8.

Do I make a test scene where the framing is similar in each lens so that you can compare resolution (ie, the texture or reference charts) and accept that the DOF will be different, or do I keep the DOF the same but vary the framing, meaning that you can't compare resolution etc.  

I'll be testing lenses that range from 8mm F4 to 200mm F4 and with apertures as much as 0.95.

In terms of what I want to shoot in the test scene, here are my notes:

• Shooting stills as they're higher resolution than video and are RAW so it eliminates most of the colour science and processing from the camera
• Controlled lighting for consistency throughout test
• Different colour temperature lights for a bit of colour (probably tungsten key with daylight WB and other lights for colour)
• Things in focus in the middle and at the edges of the frame (to test sharpness)
• Focus target in the centre
• Measuring tape running front-to-back to look at focus transitions
• Colour checker of some kind (I'll use my DIY one, not calibrated but good for comparisons)
• Fully manual settings and identical WB between shots to show colour differences
• Some bokeh with things out of focus and probably also an LED light in the background for bokeh ball
• Test every lens at every f-stop (it will be a huge number of images, but good for comparison and in my mini-test I didn't do that and I wished I had)
• I'll also shoot every lens at an identical set of settings to determine relative brightness

Thoughts?

How should I frame and is there anything missing?  (No, I don't have easy access to a model, plus I find their non-identical facial expressions taint the interpretation of images )

[mercer]

Things change when you shoot video, so I would include a 3-5 second clip with every test shot.

I’d also let the focal length speak for themselves by leaving the tripod in one locked down spot. 

Also try to do an interior and exterior version of your test. If you use a Variable ND, you can keep ur exposure similar by using the in camera meter. 

Obvious issues are the varying speeds of the lenses. To compare a wide open Voigtlander 17.5mm versus a 50mm 1.8 Nikkor wide open, the Nikkor will win every time. So it’s probably best to group your lenses by either focal length or speed... or both and have multiple groups.  

[Shirozina]

What do you hope to gain from these tests? 

[kye]

3 hours ago, mercer said:

Things change when you shoot video, so I would include a 3-5 second clip with every test shot.

I’d also let the focal length speak for themselves by leaving the tripod in one locked down spot. 

Also try to do an interior and exterior version of your test. If you use a Variable ND, you can keep ur exposure similar by using the in camera meter. 

Obvious issues are the varying speeds of the lenses. To compare a wide open Voigtlander 17.5mm versus a 50mm 1.8 Nikkor wide open, the Nikkor will win every time. So it’s probably best to group your lenses by either focal length or speed... or both and have multiple groups.  

How does video look different?  and wouldn't that be more subject to which camera you have?  If I could shoot RAW video then that might be neutral enough perhaps, but unless someone wants to send me a P4K, then....

1 hour ago, Shirozina said:

What do you hope to gain from these tests? 

Broadly, I want to see how my existing lenses (the ones I intend to keep) compare with the others, from the others I want to see which lenses I should keep (if any), I want to learn how to read an image so I can evaluate images I see online better, I want to learn more about what I like and why I like it, and I want to hear what others have to say so I can learn more about how other people see things.

I also want to give back because the internet (and all you lovely folks out there) have been crazy good to me over the years and if I'm going to all this effort then why not, and even if I wasn't feeling generous having other people point out things I didn't know to look for is well worth the work involved in sharing it anyway

[heart0less]

3 hours ago, kye said:

I also want to give back because the internet (and all you lovely folks out there) have been crazy good to me over the years

I don't want to soft-soap by any means, but it's been mostly you that have (/has?) been pointing out many useful tweaks / tricks lately, hahaha. 

I am most inspired by what you offer and from time to time try my best to help other people any way I can, too. 

It's a great thing to have you here, really. 

 

I'm not a long time member of this community, though I hope most people could agree with me. 

 

Can't really suggest any particular way to run these tests, but I'm sure the findings will prove to be a great, horizon-broadening read. 

 

Cheers! 

[mercer]

6 hours ago, kye said:

How does video look different?  and wouldn't that be more subject to which camera you have?  If I could shoot RAW video then that might be neutral enough perhaps, but unless someone wants to send me a P4K, then....

Well, in some ways, this is subjective to which camera you have... right... in the end you are going to be shooting video with a GH5.

But also remember that not all Raw is created equal. There is this misconception that all Raw is the same but you have to take into consideration the bit depth (only 12bit Raw with the GH5) the filter stack, and the sensor size.

So although you may learn more about your lenses shooting Raw stills, your end goal is video, so I would think a video test is important? 

Also you probably should find someone to shoot to see movement and skin tones.

I always liked how Nick Driftwood and Philip Bloom do their tests. They take a long ass lens into the public and get some footage. Obviously, this isn’t the same thing you’re after and in most cases they are testing cameras and not just lenses. 

So maybe if you do an elimination of your lenses, you can do a final round of tests with video at one of your favorite places to shoot the video tests?

[kye]

11 minutes ago, mercer said:

Well, in some ways, this is subjective to which camera you have... right... in the end you are going to be shooting video with a GH5.

But also remember that not all Raw is created equal. There is this misconception that all Raw is the same but you have to take into consideration the bit depth (only 12bit Raw with the GH5) the filter stack, and the sensor size.

So although you may learn more about your lenses shooting Raw stills, your end goal is video, so I would think a video test is important? 

Also you probably should find someone to shoot to see movement and skin tones.

I always liked how Nick Driftwood and Philip Bloom do their tests. They take a long ass lens into the public and get some footage. Obviously, this isn’t the same thing you’re after and in most cases they are testing cameras and not just lenses. 

So maybe if you do an elimination of your lenses, you can do a final round of tests with video at one of your favorite places to shoot the video tests?

Damn, I was hoping you wouldn't have any good points, but...

I'm not sure that movement is impacted by lenses, they all let light through in a constant stream at, well, the speed of light, so I don't think the mythical motion cadence comes from the lens.  In terms of how these things translate to video though, I may have to take a few of the lenses to a nice place and film some stuff so we can see.  

To me, if you can see lens X recording video and go wow, that's good at Y or not good at Z, then you can use the stills to kind of work out what the other lenses would be like.  I definitely don't want to record video for every lens at every aperture setting, that's for sure!!

[mercer]

23 minutes ago, kye said:

Damn, I was hoping you wouldn't have any good points, but...

I'm not sure that movement is impacted by lenses, they all let light through in a constant stream at, well, the speed of light, so I don't think the mythical motion cadence comes from the lens.  In terms of how these things translate to video though, I may have to take a few of the lenses to a nice place and film some stuff so we can see.  

To me, if you can see lens X recording video and go wow, that's good at Y or not good at Z, then you can use the stills to kind of work out what the other lenses would be like.  I definitely don't want to record video for every lens at every aperture setting, that's for sure!!

Haha. I don’t think motion cadence is mythical but that’s an argument I don’t care to engage in. Lol. 

But, I wasn’t referring to motion cadence, I was referring to motion blur and how the out of focus areas may be impacted by people passing by, birds flying by, panning, etc...

[kye]

Just now, mercer said:

Haha. I don’t think motion cadence is mythical but that’s an argument I don’t care to engage in. Lol. 

But, I wasn’t referring to motion cadence, I was referring to motion blur and how the out of focus areas may be impacted by people passing by, birds flying by, panning, etc...

LOL, I was trying to imply motion cadence doesn't exist, more that no-one seems to know where it is

With people moving in the background etc, is that a function of how the out of focus areas are rendered?  Like, bokeh balls with hard edges look great for city lights but are really distracting for everything else?  Or is it something different?

[mercer]

4 minutes ago, kye said:

LOL, I was trying to imply motion cadence doesn't exist, more that no-one seems to know where it is

With people moving in the background etc, is that a function of how the out of focus areas are rendered?  Like, bokeh balls with hard edges look great for city lights but are really distracting for everything else?  Or is it something different?

Haha, I know what you were referring to.

Let me try again because I think our signals are crossed. People moving in the background is not a function of how the out of focus areas are rendered but each individual lens will render out of focus areas differently, so people moving in the background may look worse with some lenses than others. 

[kye]

1 hour ago, mercer said:

Haha, I know what you were referring to.

Let me try again because I think our signals are crossed. People moving in the background is not a function of how the out of focus areas are rendered but each individual lens will render out of focus areas differently, so people moving in the background may look worse with some lenses than others. 

Ok, that makes sense now, I think.  But you've convinced me to shoot some video

Maybe I can put up the stills and see which ones people want to see video for.  That way I'm not carting around a whole suitcase full of lenses!

[mercer]

29 minutes ago, kye said:

Ok, that makes sense now, I think.  But you've convinced me to shoot some video

Maybe I can put up the stills and see which ones people want to see video for.  That way I'm not carting around a whole suitcase full of lenses!

That sounds like a good idea. No reason to test obvious lenses that you don’t like from your initial tests.

[thebrothersthre3]

Flaring is another thing you might want to test. That is if you care about it. 

I think your tests will yield great accurate results for the most part. 

Though do you care about the nitty gritty details that much? Would taking the lens out and shooting with it better show you if you like its characteristics. 

[no_connection]

What would be interesting to know is how the aberrations look, be it red/cyan magenta/green and so on, how it fringe around a bright light, flare, focus breathing.

And how almost in focus areas look, I don't care if the eyes are super sharp when the nose looks awful, or the transition of sharp hairs to out of focus feels wrong.

That is a 135mm f3.5 Takumar right? Focus ring it sooooo smooooooth!

[kye]

5 hours ago, thebrothersthre3 said:

Flaring is another thing you might want to test. That is if you care about it. 

I think your tests will yield great accurate results for the most part. 

Though do you care about the nitty gritty details that much? Would taking the lens out and shooting with it better show you if you like its characteristics. 

Flaring is interesting, I don't really know how that works - if it's different at different apertures or focal distances.  The frame is getting kind of crowded at this point!

5 hours ago, no_connection said:

What would be interesting to know is how the aberrations look, be it red/cyan magenta/green and so on, how it fringe around a bright light, flare, focus breathing.

And how almost in focus areas look, I don't care if the eyes are super sharp when the nose looks awful, or the transition of sharp hairs to out of focus feels wrong.

That is a 135mm f3.5 Takumar right? Focus ring it sooooo smooooooth!

The one thing that this test won't be so good for is the imperfections because most of these lenses are FF and I'm using a MFT sensor, so the lens could have dancing bears in the corners and I'd never know it.  This is where CA and the other issues tend to exist, so unfortunately this is a "lens on MFT" test.  Having said that, I do want some things in focus near the edges / corners so we can at least see how bad things are on MFT.

There should be lots of things in the slightly out-of-focus areas so I think that's covered.

Smooth like butter!!  It really is incredible, and my one is virtually brand new.  The other Takumar lenses are all just as good, if a little firmer.  It's a pity they don't look as good or have the focus ring going in the right direction ???

[no_connection]

At least my Takumar have about the same magenta fringe over the frame when wide open but goes away when stepped down.

I was just holding it up to a D810 so at 1:1 is a little less resolution than m43 at 4k.

I think many full frame lenses is going to struggle a little with raw resolution, then again 135mm on a m43 is going to be very tight and any movement or heat is going to be a thing anyway.

 

[kye]

Last lens arrived - the second Konica Hexanon 40/1.8 and it's filled with fungus... Good thing I ordered two - the other one is near perfect!  Rule of thumb for ebay - it will have fungus unless the auction states that it doesn't and the pictures are good and show that it doesn't.  Two of the ones that claimed no fungus also had fungus, one pretty badly, so buyer beware.

So, now to shoot this test.

[kye]

I've worked out a bunch of stuff about how and where to do the test.

I've decided to compare lenses so that they will have similar DoF at the same aperture and it turns out this also basically makes them the same field of view too, so that's nice.

Question - when calculating DOF (with this tool: http://www.dofmaster.com/dofjs.html) is the subject distance from the sensor-to-subject or front-of-lens-to-subject?  It gives the equations it uses here http://www.dofmaster.com/equations.html and they look pretty standard, so maybe someone knows?

This might seem like I'm nit-picking, but the distance for my 8mm is 10cm, so the length of the lens actually makes a reasonable difference.

[no_connection]

Should be front of lens to subject I think.

*edit*

Sohuld be focal length in front of sensor. Since the lens could be longer or shorter depending on how it's made.

[kye]

23 hours ago, no_connection said:

Should be front of lens to subject I think.

*edit*

Sohuld be focal length in front of sensor. Since the lens could be longer or shorter depending on how it's made.

I figured it was probably sensor distance, so I calculated it all, set it up, then the first lens I put on had too large a minimum focal distance - doh!

Still, I've shot the first round of it and will post soon