Opinion - DXOMark's camera scoring makes ZERO sense!

[Don Kotlos]

27 minutes ago, Andrew Reid said:

The problem with these charts and scores is that they don't take into account subjectivity by their very nature, they are only about technical dynamics and figures.

But then your problem is not with dxo labs testing methodology that truly offers the most objective characterization of sensors at the moment, but all the media whores that transform these results into click-bait articles. 

Of course there are always going to be people that don't understand numbers, but still no professional in their right mind will decide to use a camera based just on a score. 

[Bkn Soc of Cinematography]

I've always been skeptical about DXO ratings. They lost me with rating the A7s II lower than the A7s I for noise. Someone's out to lunch. Thanks for the heads up.

[ND64]

Here is the pixel level SNR graph for 6D vs. 5Dmark4

There is only half db difference at almost any ISO setting. 

and its DPR studio sample

Yes, 5Dmark4 is a bit noisier but not more than what 0.5 db indicates. Actually that's relatively good for its pixel size. 

[noone]

27 minutes ago, Bkn Soc of Cinematography said:

I've always been skeptical about DXO ratings. They lost me with rating the A7s II lower than the A7s I for noise. Someone's out to lunch. Thanks for the heads up.

But it doesn't measure lower for noise!

They have almost the exact same measurements for noise at the same manufacturers settings.       Yes, the settings vary slightly from each other to what they actually measured (as opposed to what the  camera says it is) but that might be down to sample variation.

It seems that for video, the A7sii may well be around stop better from the comparisons around but remember, they are testing RAW stills.

What reason would they have to lie?       The cameras use the same sensor and the results seem to indicate  that.

[aly324]

Erwin Puts, a clear-headed authority on camera and especially Leica products, has some useful words about the uses and abuses of numerical tests and rankings. http://www.imx.nl/photo/leica/camera/styled-15/

In particular he points out the questionable utility of close rankings given the inevitable margins of errors in tolerances and testing, and the wrong habit of thinking encouraged by numeral scores: that a score of 2000 vs 1000 in a test must mean camera A is "twice as good" as camera B, or that camera A "beats" camera B.

His critique that this whole industry of rankings is ultimately meant to stimulate consumer desire is definitely germane. I don't know DXOmark's exact origins, but I suspect one can trace the history of camera tests pretty closely to advertising. Puts draws comparisons with the car industry. 

Then again, we all need to live, and EOSHD is itself part of this ecology of course. We just have to be self-aware.

I really appreciate his combination of rigor about quantitative tests and critical use of them.

 

[Don Kotlos]

17 minutes ago, aly324 said:

 I don't know DXOmark's exact origins, but I suspect one can trace the history of camera tests pretty closely to advertising. 

It is indeed close to advertising, help us from the bullshit in camera advertising that is. 

DXOMark makes profit from their software that depend on these sensor characterizations. 

[jgharding]

I've never found anything in the visual arts quite so easy to ignore as DXOMARK

It's just so neckbeard and abstract!

[ADRIANPOCEA]

On 1/12/2017 at 3:54 AM, BlueBomberTurbo said:

I do agree that DXO's rankings are a bit questionable, but not too far off.  There are generally valid explanations of the issues you cited:

• NX500 over 5DS and NX1:  I've personally handled 5DSR files, and can say that the IQ is terrible. Even Canon stated not to expect much more than their old APS-C cameras in the IQ department.  I've read a few times that the NX500 is considered to have higher IQ than the NX1.  By how much, I don't know.  But viewing test RAWs of the NX1, I'd say DR and high ISO are around 1/2 stop behind the Nikon D7200.
   
• DXO One:  Its Super RAW literally is super.  It takes 4 RAW files, stacks them, and averages out the noise.  The difference is dramatic.  While the detail level isn't the best at high ISO, the lack of noise is well beyond FF capability.  This is similar to Olympus' high res RAW mode, but instead of increasing resolution, it reduces noise and increases detail at the same output size.
   
• D3X over D5:  The D5 is a bomb below ISO 1600, nearly matching the 5D III.  Even crop sensors beat it.  The sensor is tuned for mid/high ISO performance, though current technology only goes so far.  The gains, while there (+1/2 stop vs 1DX II), really aren't worth the trade off for the flexibility in low ISO RAW.  Worthy of note is that the D3X has a Sony sensor, while the D5 is Nikon's own creation.
   
• D600 over 1DX II and P40+:  It's true.  The D600 kills the 1DX II in DR at base ISO, and at worst, ties it the rest of the way up.  the 1DX II literally has years-old crop sensor performance in that area, despite Canon's massive gain in their new generation of sensors.  High ISO is also neck and neck.
  
  Vs the P40+, the sensor in the MF camera is quite old.  Despite having the resolution advantage, it loses out in DR and high ISO by quite a big margin.  By ISO 1600, colors turn to mush, which doesn't really happen on the D600 at any ISO.
   
• D3s and D700:  I've also worked with files from a D700 multiple times, and can say that yes, its sensor is outdated at this point.  It's competitive with today's crop sensor cameras (minus Canon's) at best.  The A7S/II sensor has been compared to current medium format in its DR and ability to reproduce color. 
  
  Once again, the D3s/D700's sensor is Nikon's own.  Nikon isn't very competitive when it comes to sensors, and probably had its best attempt at competing with Sony in the D4/s/f.  All of the rest of their sensors just don't stand out, though aren't as bad as Canon's.

I have a feeling that resolution plays a big part in DXO's rankings.  If you downscale the A7R II's files to A7S II size, they will certainly have an advantage in their "Sports" rating. It might also be why the A7R II beats the D810, when the D810 clearly has about 1/3 stop advantage at high ISO.  My friend tested 2x A7R IIs before returning them and keeping his D810.  #IQsnob.

For DR and high ISO, they test noise up to a certain amount.  How they get to that amount, who knows, but it's a cutoff point they chose that represents the transition from "OK" noise to offensive noise.  So while sensors may have DR response up to a certain amount of stops, after a point, it becomes wiser to turn things back a bit in software.  Where that happens is up to the user, as it's a more subjective choice.

And "Color" is more about correctly reproducing color in RAW than how the final JPG is rendered.  Color in the Canon sense is highly subjective.  Color against a known testing scene/chart isn't.

What you say it doesn't make any sense at all. You personally handled Canon 5ds files... What is that supposed to mean? You shoot with it or you didn't ? I personally handle Canon 5ds files every day, as i own one, and the IQ is awesome, and you can check that at my 500 px page, under the name Adrian Pocea. You are an armchair photographer that just bashes Canon wherever he can. I saw you on dpreview, i saw you here. Same Canon bashing crap. Even by your own logic, if you do give total credit to dxo mark, how can you rate a sensor with a score of 87 as "terrible"?  (by the way, Ken Rockwell defined it as the best dslr on the market and i agree with him). But you really hit the  bottom with this surrealistic affirmation "Even Canon stated not to expect much more than their old APS-C cameras in the IQ department"... Whattttt???? What is that supposed to mean??? So Canon came out, officially, and stated, listen, guys, don't ever expect better IQ than in our old aps-c cameras, sorry for you, we still recommend that you buy our cameras, but they suck, so , please , spend a lot of money on our cameras although we don't recommend to... This is just too funny, buddy. Get a life and get out there and start shooting

[klavaza]

These evaluations are really as flawed as those of the now defunct Hi Fidelity magazine of the 70s and 80s, but cleverly disguised as validated by high-ed tech. Thanks for sharing.