Opinion - DXOMark's camera scoring makes ZERO sense!

[Andrew Reid]

The RED Helium 8K camera (approx. APS-H size sensor) just leapt to the top of the DXOMark charts with a score of 108.

Read the full article

[webrunner5]

Man that outfit is prettier than a Speckled Puppy!

[Andrew Reid]

Then there is the curious case of the missing Pentax 645Z score

http://pentaxrumors.com/2015/12/25/pentax-645z-scored-101-at-dxomark/

Missing in action!

3 minutes ago, webrunner5 said:

Man that outfit is prettier than a Speckled Puppy!

Yes it is.

I want all my cameras to be IMAX white from now on.

[Henry Gentles]

Tony Northrup who is a Programmer/Science background has plenty to say about DX0 ranking system. Color Rendition is based on how they feel about it apparently ie not any objective data etc. He said the same, DX0 ranking is deeply flawed from a technical standpoint!

[noone]

DXO has their explanations.

https://www.dxomark.com/About/Sensor-scores/Overall-Score

 

https://www.dxomark.com/About/Sensor-scores/Use-Case-Scores

Low light score in particular is an actual ISO and I think the easiest to explain.

"Sports & action photography: Low-Light ISO

Unlike the two previous scenarios in which light is either generous (studio) or stability is assured (landscape), photojournalists and action photographers often struggle with low available light and high motion. Achieving usable image quality is often difficult when pushing ISO.

When shooting a moving scene such as a sports event, action photographers’ primary objective is to freeze the motion, giving priority to short exposure time. To compensate for the lack of exposure, they have to increase the ISO setting, which means the SNR will decrease. How far can they go while keeping decent quality? Our low-light ISO metric will tell them.

The SNR indicates how much noise is present in an image compared to the actual information (signal). The higher the SNR value, the better the image looks, because details aren't drowned by noise. SNR strength is given in dB, which is a logarithmic scale: an increase of 6 dB corresponds to doubling the SNR, which equates to half the noise for the same signal.

An SNR value of 30dB means excellent image quality. Thus low-light ISO is the highest ISO setting for a camera that allows it to achieve an SNR of 30dB while keeping a good dynamic range of 9 EVs and a color depth of 18bits.

A difference in low-light ISO of 25% represents 1/3 EV and is only slightly noticeable.

As cameras improve, low-light ISO will continuously increase, making this scale open."

 

It has nothing to do with AF etc so I think the sports scores are pretty reasonable maybe more so if you just consider them as low light rather than sports and remember it is based on their criteria.       I am fine with using an A7s for night time sports.      I use an old manual focus 300 2.8 anyway so it just means I can use a higher shutter speed.     None of the shots would be printed huge so 12mp is fine.

A modern FF DSLR would be a better sports CAMERA most of the time but does it have a better sports SENSOR (given DXOmarks criteria)?

The overall scores are a bit based on voodoo as the bits that go into that are subjective without full explanation as to weighting.

I think some of the anomalies might be because of a low number of samples tested given many cameras get slightly different scores with the same sensors.      A slight difference might be just enough to take a camera a bit over or a bit under their marks.     I don't think the A7s is any noisier than the A7sii and It seems the colour depth might be why the A7sii gets a lower score for low light (the point they cross 18 bits for colour sensitivity).

For video it is all a bit silly though as they are only testing RAW stills and most video is Jpeg.

I would love to see a site test sensors for video.    RAW and otherwise.

 

[anax276]

Thank you for this article! These scores often left me wondering what exactly is going on... 

That's one of the reasons I prefer real-world tests/experiences (like the ones at EOSHD) over obscure scores, comparisons of technical specifications and theoretical discussions about cameras by people not actually using them.

[BlueBomberTurbo]

I do agree that DXO's rankings are a bit questionable, but not too far off.  There are generally valid explanations of the issues you cited:

• NX500 over 5DS and NX1:  I've personally handled 5DSR files, and can say that the IQ is terrible. Even Canon stated not to expect much more than their old APS-C cameras in the IQ department.  I've read a few times that the NX500 is considered to have higher IQ than the NX1.  By how much, I don't know.  But viewing test RAWs of the NX1, I'd say DR and high ISO are around 1/2 stop behind the Nikon D7200.
   
• DXO One:  Its Super RAW literally is super.  It takes 4 RAW files, stacks them, and averages out the noise.  The difference is dramatic.  While the detail level isn't the best at high ISO, the lack of noise is well beyond FF capability.  This is similar to Olympus' high res RAW mode, but instead of increasing resolution, it reduces noise and increases detail at the same output size.
   
• D3X over D5:  The D5 is a bomb below ISO 1600, nearly matching the 5D III.  Even crop sensors beat it.  The sensor is tuned for mid/high ISO performance, though current technology only goes so far.  The gains, while there (+1/2 stop vs 1DX II), really aren't worth the trade off for the flexibility in low ISO RAW.  Worthy of note is that the D3X has a Sony sensor, while the D5 is Nikon's own creation.
   
• D600 over 1DX II and P40+:  It's true.  The D600 kills the 1DX II in DR at base ISO, and at worst, ties it the rest of the way up.  the 1DX II literally has years-old crop sensor performance in that area, despite Canon's massive gain in their new generation of sensors.  High ISO is also neck and neck.
  
  Vs the P40+, the sensor in the MF camera is quite old.  Despite having the resolution advantage, it loses out in DR and high ISO by quite a big margin.  By ISO 1600, colors turn to mush, which doesn't really happen on the D600 at any ISO.
   
• D3s and D700:  I've also worked with files from a D700 multiple times, and can say that yes, its sensor is outdated at this point.  It's competitive with today's crop sensor cameras (minus Canon's) at best.  The A7S/II sensor has been compared to current medium format in its DR and ability to reproduce color. 
  
  Once again, the D3s/D700's sensor is Nikon's own.  Nikon isn't very competitive when it comes to sensors, and probably had its best attempt at competing with Sony in the D4/s/f.  All of the rest of their sensors just don't stand out, though aren't as bad as Canon's.

I have a feeling that resolution plays a big part in DXO's rankings.  If you downscale the A7R II's files to A7S II size, they will certainly have an advantage in their "Sports" rating. It might also be why the A7R II beats the D810, when the D810 clearly has about 1/3 stop advantage at high ISO.  My friend tested 2x A7R IIs before returning them and keeping his D810.  #IQsnob.

For DR and high ISO, they test noise up to a certain amount.  How they get to that amount, who knows, but it's a cutoff point they chose that represents the transition from "OK" noise to offensive noise.  So while sensors may have DR response up to a certain amount of stops, after a point, it becomes wiser to turn things back a bit in software.  Where that happens is up to the user, as it's a more subjective choice.

And "Color" is more about correctly reproducing color in RAW than how the final JPG is rendered.  Color in the Canon sense is highly subjective.  Color against a known testing scene/chart isn't.

[tomekk]

49 minutes ago, BlueBomberTurbo said:

I do agree that DXO's rankings are a bit questionable, but not too far off.  There are generally valid explanations of the issues you cited:

• NX500 over 5DS and NX1:  I've personally handled 5DSR files, and can say that the IQ is terrible. Even Canon stated not to expect much more than their old APS-C cameras in the IQ department.  I've read a few times that the NX500 is considered to have higher IQ than the NX1.  By how much, I don't know.  But viewing test RAWs of the NX1, I'd say DR and high ISO are around 1/2 stop behind the Nikon D7200.
   
• DXO One:  Its Super RAW literally is super.  It takes 4 RAW files, stacks them, and averages out the noise.  The difference is dramatic.  While the detail level isn't the best at high ISO, the lack of noise is well beyond FF capability.  This is similar to Olympus' high res RAW mode, but instead of increasing resolution, it reduces noise and increases detail at the same output size.
   
• D3X over D5:  The D5 is a bomb below ISO 1600, nearly matching the 5D III.  Even crop sensors beat it.  The sensor is tuned for mid/high ISO performance, though current technology only goes so far.  The gains, while there (+1/2 stop vs 1DX II), really aren't worth the trade off for the flexibility in low ISO RAW.  Worthy of note is that the D3X has a Sony sensor, while the D5 is Nikon's own creation.
   
• D600 over 1DX II and P40+:  It's true.  The D600 kills the 1DX II in DR at base ISO, and at worst, ties it the rest of the way up.  the 1DX II literally has years-old crop sensor performance in that area, despite Canon's massive gain in their new generation of sensors.  High ISO is also neck and neck.
  
  Vs the P40+, the sensor in the MF camera is quite old.  Despite having the resolution advantage, it loses out in DR and high ISO by quite a big margin.  By ISO 1600, colors turn to mush, which doesn't really happen on the D600 at any ISO.
   
• D3s and D700:  I've also worked with files from a D700 multiple times, and can say that yes, its sensor is outdated at this point.  It's competitive with today's crop sensor cameras (minus Canon's) at best.  The A7S/II sensor has been compared to current medium format in its DR and ability to reproduce color. 
  
  Once again, the D3s/D700's sensor is Nikon's own.  Nikon isn't very competitive when it comes to sensors, and probably had its best attempt at competing with Sony in the D4/s/f.  All of the rest of their sensors just don't stand out, though aren't as bad as Canon's.

I have a feeling that resolution plays a big part in DXO's rankings.  If you downscale the A7R II's files to A7S II size, they will certainly have an advantage in their "Sports" rating. It might also be why the A7R II beats the D810, when the D810 clearly has about 1/3 stop advantage at high ISO.  My friend tested 2x A7R IIs before returning them and keeping his D810.  #IQsnob.

For DR and high ISO, they test noise up to a certain amount.  How they get to that amount, who knows, but it's a cutoff point they chose that represents the transition from "OK" noise to offensive noise.  So while sensors may have DR response up to a certain amount of stops, after a point, it becomes wiser to turn things back a bit in software.  Where that happens is up to the user, as it's a more subjective choice.

And "Color" is more about correctly reproducing color in RAW than how the final JPG is rendered.  Color in the Canon sense is highly subjective.  Color against a known testing scene/chart isn't.

I think, overall score is pretty much meaningless and comparisons between some models are pointless. It's like arguing about cars:
Me: My sports car has V12 5L engine, rear wheel drive, 700BHPs, 2 seats.
You: ok, but my family car has 7 seats, 4x more space, better audio system,more safety features, all wheel drive, mpg is 5x better, emissions are 10x lower, it's 20x cheaper, 50x cheaper to maintain... and it's much quieter because it's a hybrid. Therefore, you clearly overpaid for a worse car because you scored less overall ! You can't even take family out for a picnic, hellooooo!!! 

 

[Oliver Daniel]

I don't bother looking at charts. Just try out a few cameras, see what looks best, and shoot with it. 

[bigfoot]

Dxo marks always been like that. 

It's not THE reference ..it's only a reference for those who actually know how to read the graphs and analyse some of the results. Overall score means as much as saying that the sky is blue. 

[gnugent]

one thing to note here there reason arri and sony F5 etc don't get included in this ranking, is as far as i know they don't have a dedicated stills mode 

i shot have stills on a dragon before and they are pretty good and its quite easy to switch to that mode during a shoot

 also the i would agree with the DR on the helium it,s very good  as good as my K1 as far as i can see 

and what makes it even better that's from a moving image 

i,m sure you could pull an individual .ari or DNG from one of these other cameras and test ....but thats does,nt seem to be how the DXO guys work.

i,m glad some one brought up the 645z ...it is quite strange that was left out ...that is truly brillant camera 

 

 

[cantsin]

Quote from the EOSHD op-ed article: "DXOMark claims to measure RAW sensor performance but actually some sensors process the RAW data in special ways."

The DxO review of the RED Helium sensor does actually acknowledge this and contains a respective disclaimer:

"Whatever noise reduction system RED employs creating the RAW images from the Helium sensor, its presence means that we aren’t measuring just the RED sensor, so its results aren’t directly comparable to those from camera sensors we have tested from other vendors, whose RAW results come straight from the sensor with no prior noise reduction processing."

https://www.dxomark.com/Reviews/RED-Helium-8K-DxOMark-Sensor-Score-108-A-new-all-time-high-score2

[hmcindie]

If you use noise reduction your low light score will jump up crazily because the SNR goes down as the image smooths.

I realized that DXomark was kinda weird when I had the Canon 7d and the Sony nex-5n. The nex-5n scored much better but I got better images with the 7d. Also the differences between them were extremely marginal but in scores they were miles apart. Also the nex-5n had trouble with lighting that was  easier for the 7d (even in raw, especially blue highlights from leds) which DXOmark didn't spot at all.

[DBounce]

I also feel that unless shooting charts is your thing, these test are not so relevant. But as a tool to tell your story, we'll what's not to love about the Red's image? Barring physical or practical restrictions I would pick the Red all day long.  And that's even assuming the same price.

[wolf33d]

4 hours ago, BlueBomberTurbo said:

I do agree that DXO's rankings are a bit questionable, but not too far off.  There are generally valid explanations of the issues you cited:

• NX500 over 5DS and NX1:  I've personally handled 5DSR files, and can say that the IQ is terrible. Even Canon stated not to expect much more than their old APS-C cameras in the IQ department.  I've read a few times that the NX500 is considered to have higher IQ than the NX1.  By how much, I don't know.  But viewing test RAWs of the NX1, I'd say DR and high ISO are around 1/2 stop behind the Nikon D7200.
   
• DXO One:  Its Super RAW literally is super.  It takes 4 RAW files, stacks them, and averages out the noise.  The difference is dramatic.  While the detail level isn't the best at high ISO, the lack of noise is well beyond FF capability.  This is similar to Olympus' high res RAW mode, but instead of increasing resolution, it reduces noise and increases detail at the same output size.
   
• D3X over D5:  The D5 is a bomb below ISO 1600, nearly matching the 5D III.  Even crop sensors beat it.  The sensor is tuned for mid/high ISO performance, though current technology only goes so far.  The gains, while there (+1/2 stop vs 1DX II), really aren't worth the trade off for the flexibility in low ISO RAW.  Worthy of note is that the D3X has a Sony sensor, while the D5 is Nikon's own creation.
   
• D600 over 1DX II and P40+:  It's true.  The D600 kills the 1DX II in DR at base ISO, and at worst, ties it the rest of the way up.  the 1DX II literally has years-old crop sensor performance in that area, despite Canon's massive gain in their new generation of sensors.  High ISO is also neck and neck.
  
  Vs the P40+, the sensor in the MF camera is quite old.  Despite having the resolution advantage, it loses out in DR and high ISO by quite a big margin.  By ISO 1600, colors turn to mush, which doesn't really happen on the D600 at any ISO.
   
• D3s and D700:  I've also worked with files from a D700 multiple times, and can say that yes, its sensor is outdated at this point.  It's competitive with today's crop sensor cameras (minus Canon's) at best.  The A7S/II sensor has been compared to current medium format in its DR and ability to reproduce color. 
  
  Once again, the D3s/D700's sensor is Nikon's own.  Nikon isn't very competitive when it comes to sensors, and probably had its best attempt at competing with Sony in the D4/s/f.  All of the rest of their sensors just don't stand out, though aren't as bad as Canon's.

I have a feeling that resolution plays a big part in DXO's rankings.  If you downscale the A7R II's files to A7S II size, they will certainly have an advantage in their "Sports" rating. It might also be why the A7R II beats the D810, when the D810 clearly has about 1/3 stop advantage at high ISO.  My friend tested 2x A7R IIs before returning them and keeping his D810.  #IQsnob.

For DR and high ISO, they test noise up to a certain amount.  How they get to that amount, who knows, but it's a cutoff point they chose that represents the transition from "OK" noise to offensive noise.  So while sensors may have DR response up to a certain amount of stops, after a point, it becomes wiser to turn things back a bit in software.  Where that happens is up to the user, as it's a more subjective choice.

And "Color" is more about correctly reproducing color in RAW than how the final JPG is rendered.  Color in the Canon sense is highly subjective.  Color against a known testing scene/chart isn't.

+10000

And kudos to DXO for giving us this. At least we have a good understanding of performance sensor. I verified their measurement in real world with every camera I owned. 

Cross their results with the Dpreview image comparison tool for ISO and DR and it matches well. 

I don't understand your frustration Andrew.

[webrunner5]

Well even if their testing is flawed to some people, it is flawed for all cameras tested, so it is a fair comparison to each one, they all use the same technique applied to them.

So maybe the outcome is maybe not true reality, the order they come out is probably what real life output will be compared to each other camera tested.

Red and Arri have always have crazy high DR compared to others brands. And it has proven to be true.

[Don Kotlos]

7 hours ago, BlueBomberTurbo said:

I do agree that DXO's rankings are a bit questionable, but not too far off.  

Yep in my experience with various cameras thats how I see it too. Its unfortunate that they don't post their margins of error though. Also video performance is another deal altogether, and their metrics have little to do with that. 

Quote

I have a feeling that resolution plays a big part in DXO's rankings.  If you downscale the A7R II's files to A7S II size, they will certainly have an advantage in their "Sports" rating. It might also be why the A7R II beats the D810, when the D810 clearly has about 1/3 stop advantage at high ISO.  My friend tested 2x A7R IIs before returning them and keeping his D810.  #IQsnob.

DXO provides both pixel level and constant image size measurements ("screen" & "print" respectively). The final score is calculated from the print scores (as it should be) but many people do their comparisons when looking at the pixel level. Moreover, many times people just look at the SNR18% as a single metric for ISO performance but that is not the only thing that matters. Dynamic range & color sensitivity play a big part as well for the final "Sports" metric (very bad name choice by the way). As an example, I care more about color reproduction & tonal range at higher ISOs than noise or dynamic range, and after using both the D800 and A7rii extensively my experience matches closely their metric scores. 

Quote

And "Color" is more about correctly reproducing color in RAW than how the final JPG is rendered.  Color in the Canon sense is highly subjective.  Color against a known testing scene/chart isn't.

I see two color aspects when dealing with RAW files

• Color sensitivity (how much information is in the file)
• Color accuracy (reproduction & interpretation)

I believe the DXO color sensitivity scores try to address the first point but of course are influenced by the second (which they have tried to standardize but some sensors like x-trans require different standards and thus remain untested). 

An example: The raw camera profile in Lightroom for A7rii is fairly bad for my taste, and much worse than the Nikon (which might explain your friends decision by the way). Once you change the RAW interpretation to your personal tastes, the A7rii sensor (or any sensor for that matter) can show its real potential. I made the following post for that exact reason:

 

5 hours ago, Oliver Daniel said:

I don't bother looking at charts. Just try out a few cameras, see what looks best, and shoot with it. 

Yes yes yes.

Scores should be used only when you don't have the choice of testing different cameras, and not to make you feel uneasy about your equipment.

If you like the output, why should a number should change that? 

[MattH]

It's all a load of hot air.   Read noise is all they need to measure.

[Don Kotlos]

4 minutes ago, MattH said:

It's all a load of hot air.   Read noise is all they need to measure.

At different gains  

[cantsin]

Here's a perhaps unorthodox take on the matter: DxO scores are really useful if you use DxO Optics Pro as your RAW converter, because the DxO lab measurements provide the basis for the camera profiles of the application.

I happen to be a person who buys a stills camera as a hardware frontend for DxO Optics Pro (hence, no Fuji X-Trans for me...), just as I buy video cameras as hardware frontends for Resolve. If this is your usage scenario, then DxOMark is really useful. Since the internal color science of cameras doesn't matter with DxO Optics Pro (+ FilmPack), you can indeed buy a 2nd hand NEX-5N for $150 and obtain better stills image quality than from a $1500 Canon 7D Mark II: https://www.dxomark.com/Cameras/Compare/Side-by-side/Canon-EOS-7D-Mark-II-versus-Sony-NEX-5N___977_737

That these results won't necessarily translate to Lightroom or other Raw converters - and don't say anything about out-of-cam JPEG quality -, isn't surprising.