Do specifications mean anything regarding cameras' performance? A research.

[Guest Ebrahim Saadawi]

Often we see camera releases, where the product gets complete acceptance or refusal based on the specification list provided alongside the announcement. The camera is evaluated entirely as good or bad purely based on that list. In fact, it's not only on release of new product, it's a practice conducted throughout the product's life, where people compare it to other cameras based on the specification list, even make full reviews and conclusions.

Is it possible to evaluate a product that way? That's the question we're trying to answer. The only way to do so is to look at

1-Whether spec sheet provide enough data

2-The examples in previous and current product quality vs. their spec sheets. 

 

Specification sheets of cameras provide following information

 

1 -Resolution, stated in a number of 1080p, 4K, etc.

Is that number indicative of the camera's resolution/sharpness? The answer is no. The spec data about resolution merely states the number pixels in the final file. Of course we all know that's not an indicative of final image resolution. Canon DSLRs have a spec. of 1080p, while Canon C300 has a spec resolution on 1080p, yet the difference in resolution is absolutely enormous. The reason for that is, there are many other elements that determine the image resolution that are not mentioned in the spec sheet resolution section. These are,

-how many sensor pixels is that image derived from? Is it derived from the same number of pixels without down-sampling? In this case we know as a Bayer sensor this means the actual resolution is 60% of that number stated in the spec.

Or is it derived from a higher resolution part of a sensor? If it is then the second element that determines resolution is

-The down-scaling method. When the image is derived from a larger number of pixels, how is it actually down-sampled? Is it line-skipping rows of the sensor? is it pixel-binning? is it reading the entire area and down-sampling, and if so, what algorithm is used, Cubic, lanczos, a bad one that results in lowered resolution and aliasing artifacts?

-The lens resolving power: in case of fixed lens cameras, or interchangeable cameras when the user intends to use a specific lens. 

Therefore based on that, a resolution of a video cameras cannot be evaluated reading the spec sheet, as it simply does not provide the needed data to do so. The 1080p, 4K, number in the spec sheet is meaningless and does not determine how the resolution of the camera is going to be like.

 

2- Low-light performance: only stated as minimum and maximum ISO values and not the noise amount or noise pattern at different ISOs, making this section completely useless.

 

3- Dynamic range: is always in fact not stated at all, but sometimes it is, in a number of stops. The dynamic range number hugely varies among tester and users, and has various definitions, for examples a company can measure the DR of a camera as the number of stops between the points where the camera clips at both ends, yet this does not mean anything to the end user as the quality of the areas near both ends are not stated. The shadow areas can contain detail (not clip) but have significant amount of noise and compression artifacts making a significant decrease in the final DR of the camera to the operator. Plus the highlight roll-off and the shadow roll off are not stated. Making the DR number in the spec sheet nearly a meaningless indication of the final range the camera will be able to output.

 

4- Colour performance: only stated in bit depth and chroma sub-sampling. In fact, the most important aspect in cameras' colour rendition is the colour science. It's a processing method, a complicated number of code lines that change and shift how each colour seen by the camera is represented in the final image. Each company, and each camera within each company can have a different colour rendition, something not stated in any spec sheets, making it impossible to judge how pleasing or un-pleasing the colours produced by the camera are.

 

5- Lens: In fixed lens cameras and in lens specifications of a separate lens.

The stated specs are merely the following, the focal length, the aperture values, and the size and weight of the lens. However, nothing about the lens optical performance is stated. A lens performance is a very complicated subject and it's silly one could evaluate it using the focal length and aperture values. These ''only'' state what the zoom range is, and the DOF possible. Nothing else is stated. How the resolution is, how the resolution is at different focal lengths, how the resolution is in the corners vs center, how the out-of-focus rendition looks, how much chromatic aberrations exist, colour fringing, focus breathing, Coma, Contrast, peripheral shading, nothing is mentioned. In fact, even the focal lengths mentioned are not entirely accurate on most lenses, and even the aperture values do not indicate the actual brightness (T value). Therefore a lens simply cannot be evaluated based on its spec list. The Canon 50mm F/1.4 is on specs an identical lens to the zeiss Otus 55mm F/1.4. An example, The 24-240mm F/2.8 lens on the RX10 is only better than the XC10 24-300mm F/3.5-5.6 in aperture value (DOF), yet everything else (about a dozen factors in lenses performance) about the lenses are not known and must be evaluated first, yet people on announcement crowned the 24-240mm F/2.8 the ''better'' lens ultimately, not knowing it could be a much better or a much worse lens. Specs simply don't give enough data, actually in case of lenses, almost none.

 

6- Codec, is always stated next, as H.264, ProRes, along with the bitrate and bitdepth and chroma subsampling rate.

That is accurate and it does state the compression method used in the camera, but what's important to know is that it does not indicate in anyway how the quality of the image and codec are going to be. That's determined by the implementation of the actual codec in the specific camera, and the compression quality is determined by many other elements not stated in the spec list, such as the actual resolution going into the codec, the downsampling method, the processor performance, the colour dye and colour science within the camera, and the final image enhancing algorithms. Prominent examples are a camera like the Canon C100 which has a 24mbps H.264 4:2:0 8bit codec, yet has a strikingly better compression performance than a 550D with an identical on paper 24mbps 4:2:0 8bit H.264 codec. Therefore the codec numbers on the spec sheet mean very little to almost none in the final quality of the image and even in the final compression performance, as in how hard the image can be pushed in post. The elements that do mean something are simply not stated in specification lists.

 

7- Frame rates, are always stated next

Yes they do give an accurate representation of the camera's maximum frame rate ability. But just that, all the other more important data about that mode are not stated. The downsides implemented to achieve these higher frame rates are not mentioned, for most cameras they achieve it through skipping lines, which results it horrible aliasing and a significant resolution less at higher frame rates, even on high end cinema cameras. Therefore, when the spec sheet mentions a camera can do 1080p at 25p and 50p, it's true, but whether the 50p mode is usable in quality is an entirely different subject again, not stated in the spec sheet. So it's not the frame rate number is not indicative of the camera's capability as a slow-motion shooter.

 

8- Written next can be the Audio specs.

What's mentioned is the format, hz number, the type of input/output. However, the most important elements are not stated. The type and quality of the Pre-amps in the camera, the actual sound quality, whether the audio can be controlled manually or not, therefore one simply cannot get any indication of the camera's capability to record sound based on the spec sheet, it's almost a useless section of the sheet.

 

9- Battery:

Is mentioned in the spec list as the product name, as an LP-E6n or an XD-T62 or a WTF-876 and sometimes it mentions a capacity number. However, the spec list does not mention the power draw of the camera whilst shooting video, it doesn't give any indications whatsoever on how good or bad the battery life in the camera is.

 

10- There are many very vital elements in cameras that are not mentioned altogether.

-Aliasing performance 
-Moire performance 
-Rolling shutter amount 
-Highlight roll-off 
-Resolution & quality of crop modes 
-LCD & EVF quality and performance under day light 
-The menu system speed 
-The menu system user interface 
-The responsiveness of buttons and dials 
-The feel of the camera in hand 
-The ergonomics of the handling of the camera 
-The heat management 
-The durability under various conditions 
-The materials used in the built

 

11- The elements that ARE accurate and mentioned in spec sheets are actually:

-The size of the camera 
-The size of the sensor
-The number of pixels on the total sensor
-The weight 
-The media slots
-The LCD size 
-The total number of pixels on the sensor 
-The price of the camera

That's it.

If you want to review a camera based on that be my guest it's fine, but it's just a physical impossibility to know whether this camera is ''good'' or ''bad'' when you have absolutely no idea about the 11 elements of image quality it produces, not even one including resolution.

This was the answer to the question about whether spec sheets provides enough data and the answer is a clear no.

 

Next time there's a new announcement, please think about the meaning of the represented data and whether they can be used to evaluate the camera. A camera is way more than just a sheet from the maker, that sheet only presents about 10% of the quality of the final product. Please, don't buy a camera based on specs, but it based on the quality of the camera's image to you, and the quality of the design, nothing else, and this data can only be obtained by using and testing the camera heavily, or by reading the review of someone you trust who does that.

 

Let's look at a recent spec-impressive announcement which we all crowned a wonderful camera just based on the list, The Sony A7RII. 

We're excited about the 4K resolution, but do we know how much resolution it's going to actually resolve?

in FF or s35 modes?

Do we know whether it has deadly aliasing/moire or not?

do we know if it will have deal-braking rolling shutter?

do we know if it will record more than 10 minutes without over heating?

Do we know how the low-light performance is?

Do we have any information of the colour science?

on the amount of DR?

on the Highlight roll-off?

on the noise pattern and texture?

do we know if the battery is going to last more than half an hour with a 42mp 4K sensor?

Do we know if the 5 axis stabilization will give acceptable results in video mode?

Do we know if it won't have unacceptable audio quality?

Do we know how the still images are going to look?

the compression in RAW and JPEG codecs and whether they're horrible compression methods or not?

Do we know how long it is going to be process and clear the buffer? Do we know the start-up time?

Do we know whether the camera will be broken instantly under rain? Dust? High temperature?

Do we know if the slow-motion will have SD resolution and unacceptable aliasing?

Do we know if the video AF works well?

Do we know if the codec won't have enormous banding, macro-blocking, mosquito noise, edge pixelation issues?

Do we know anything about the camera?

We don't,

yet 99% of the PR readers came out believing firmly it's a great camera, without having any data to support it.

(The A7RII might do extremely well in ALL these areas but the point is made I believe)

 

Specs only mean the size and weight of the camera. Wait, in fact, the listed size and weight are always misleading as it's stated without a battery, cards, handles, lens, microphone, oh well.

Just please stop reviewing camera on spec-sheets, it's only harming the film-making community and leading people to choose incorrect products. 

[agolex]

Well, most reviews are just part of the marketing machine and I don't see a way of changing that. Nice write-up anyhow!

[Oliver Daniel]

Thanks for sharing. 

I think the bulk of that relies on reviewers with a lot of time on their hands/sponsor paid to share their thoughts. A camera company would never go into such detail, as this will highlight weaknesses to potential buyers. The best way with any new technology is to test it yourself and see what works for you. 

I see you mentioned the RX10 II vs the Canon XC10. Many people laughed at the Canon release....although the RX10 II has a far more spectacular looking spec sheet, the image quality looks inferior to that of the XC10 (from what I've seen so far), and that's where the Canon camera has it's value and higher price tag. Still, I'd probs buy the RX10 II because it has features that fit my clients better. 

To be honest, I don't get too worked up about these technical things. I just shoot. No amount of things like shadow noise will prevent that, althought it's useful to know 

[Mattias Burling]

"- The durability under various conditions 
-The materials used in the built"

These two I find are often mentmentioned in the prosumer to pro segment as well as if its one of the main features of the camera. 

[MattH]

Whoa! that is one hell of a post.  Basically, you have intellectualised the thought process of a rational intelligent person in this field.  The problem is that not all people are sufficiently intelligent and also sufficiently rational.  This is why spec lists work as advertising. This is what has driven the megapixel war and now the 4k push.  What use is 4k if its got aliasing, macroblocking, focus pulsating and rolling shutter skew to the eyeballs?  Try pointing that out to someone who has already decided to buy the camera because its spec sheet says its 4k and has this and that, and they will say "Dude! this file is COMPRESSED! You need to see the original."  Yeah, because no video on youtube has ever looked good, right?  It's very hard if not impossible to get through to these people.  But be thankful that you are able to evaluate things rationally.

[maxotics]

To follow onto what Mattias said, video is still secondary to these consumer cameras.  Specs are more focused on stills, like dynamic range and ISO.  If you want meaningful specs you need to get the technology focused on that use.  I remember when I first came to this blog and Andy was 24/7 raving about the Panasonic G6.  I didn't understand.  The Sony and Canon cameras I had at the time seemed just as good and had the same specs.  Then I bought a used G5 and I saw what he was saying.  Lots of intangibles.  Like autofocus, how do you really spec that?  I compared a GH4 against the A6000 and only then could I see the difference.  I have yet to use a camera (and I've tried most) that doesn't have some strength over another camera (and weakness).  

I think the question you're driving at Ebrahim, is what specs do we need to make choosing cameras betters?  I'd love that too.  Maybe Andrew can come up with something.

[Ed_David]

This is incredible. Thank you for writing this. Yes puts it all into perspective. Cameras should never be preordered. Only ordered after proper tests are made. The whole preorder game is not smart.

Also waiting for firmware to come up to speed. Better to wait 6 to 12 months after a cameras release.

 

Great article ebrahim!!

[fuzzynormal]

Not all people are sufficiently intelligent and also sufficiently rational.

​Definitely not.  

Brand loyalty, a strong hard-wired-to-the-brain-physiological-effect that marketers have been successfully exploiting for generations gets in the way of rationality.  Humans are irrational people when willingly or unwillingly ignorant --and they rely on those "gut-decisions," typically based on familiarity, in those instances.  Advertisers know this and it's why advertisements/marketing is most often built not to actively sell product but to get the familiarity of the brand stuck in your mind in a welcoming way.  After all, Coca-Cola ads never sell the actual thing, they sell happiness, love, and comfort.  And to get it you're encouraged to buy the can/bottle with their logo on it.  Same with imaging.  You want to be a great photographer you HAVE to buy a" Canikon" ...because that's what just about everybody has familiarity with.  You heard about it, you know people that use it, etc.

So, as us filmmakers/photographers get more informed we can make more rational decisions.  Newbies can't really do this as they don't grasp the details as well, so they'll most likely tend toward Canon and Nikon unless actively exposed to different ideas.

This post is an excellent example of explaining those different ideas in the context of making motion pictures.  It might seem redundant to us that have been in the market for awhile, but for somebody new to it, it'll be very helpful just as a way to understand how to consider things they've never considered.

Good job on articulating those basics Ebrahim, it'll definitely help someone in the future!  If I could up vote this or pin it to the top of the page, I would absolutely do so!

[Sekhar]

This a great checklist Ebrahim, and thanks for putting it together. Unfortunately it is also of mostly of just academic interest and is useful for reference rather than as a practical guide to making buying decisions. We're human beings and (rightly) prioritize on a few key aspects that matter to us. Surely nobody on this planet finds a partner based on a 128-point checklist even if it's probably the most important decision they'll make in their life? And several of the points in the list are nice and convenient to state but next to impossible to verify. I mean what on earth does "durability under various conditions" mean? And in any case how will you verify that for a hot camera that is just out?

[Ed_David]

Man tough crowd

[fuzzynormal]

just academic interest and is useful for reference rather than as a practical guide to making buying decisions.

​Couldn't disagree more.  I think it's a useful guide for an upcoming purchase.  Especially for the uninitiated.  If you read the entire post you'll see it actually advises considerations for "hot" cameras.  His conclusions in this regard on on target, imho.

[agolex]

Different people, different decision making, eh. I'd never follow any guide on what to purchase, but there sure is lots of people who do, especially many of the more academically minded. As always, we're just patting shoulders or bashing skulls, so let's keep going.

[Guest Ebrahim Saadawi]

I never wrote an article on which everyone agrees, that's great, and even well, boring if I may  

anyone? 

specs don't mean absolutely nothing, so let's discuss that, what do they actually give?

[sanveer]

Often we see camera releases, where the product gets complete acceptance or refusal based on the specification list provided alongside the announcement. The camera is evaluated entirely as good or bad purely based on that list. In fact, it's not only on release of new product, it's a practice conducted throughout the product's life, where people compare it to other cameras based on the specification list, even make full reviews and conclusions.

Is it possible to evaluate a product that way? That's the question we're trying to answer. The only way to do so is to look at

1-Whether spec sheet provide enough data

2-The examples in previous and current product quality vs. their spec sheets. 

 

Specification sheets of cameras provide following information

 

1 -Resolution, stated in a number of 1080p, 4K, etc.

Is that number indicative of the camera's resolution/sharpness? The answer is no. The spec data about resolution merely states the number pixels in the final file. Of course we all know that's not an indicative of final image resolution. Canon DSLRs have a spec. of 1080p, while Canon C300 has a spec resolution on 1080p, yet the difference in resolution is absolutely enormous. The reason for that is, there are many other elements that determine the image resolution that are not mentioned in the spec sheet resolution section. These are,

-how many sensor pixels is that image derived from? Is it derived from the same number of pixels without down-sampling? In this case we know as a Bayer sensor this means the actual resolution is 60% of that number stated in the spec.

Or is it derived from a higher resolution part of a sensor? If it is then the second element that determines resolution is

-The down-scaling method. When the image is derived from a larger number of pixels, how is it actually down-sampled? Is it line-skipping rows of the sensor? is it pixel-binning? is it reading the entire area and down-sampling, and if so, what algorithm is used, Cubic, lanczos, a bad one that results in lowered resolution and aliasing artifacts?

-The lens resolving power: in case of fixed lens cameras, or interchangeable cameras when the user intends to use a specific lens. 

Therefore based on that, a resolution of a video cameras cannot be evaluated reading the spec sheet, as it simply does not provide the needed data to do so. The 1080p, 4K, number in the spec sheet is meaningless and does not determine how the resolution of the camera is going to be like.

 

2- Low-light performance: only stated as minimum and maximum ISO values and not the noise amount or noise pattern at different ISOs, making this section completely useless.

 

3- Dynamic range: is always in fact not stated at all, but sometimes it is, in a number of stops. The dynamic range number hugely varies among tester and users, and has various definitions, for examples a company can measure the DR of a camera as the number of stops between the points where the camera clips at both ends, yet this does not mean anything to the end user as the quality of the areas near both ends are not stated. The shadow areas can contain detail (not clip) but have significant amount of noise and compression artifacts making a significant decrease in the final DR of the camera to the operator. Plus the highlight roll-off and the shadow roll off are not stated. Making the DR number in the spec sheet nearly a meaningless indication of the final range the camera will be able to output.

 

4- Colour performance: only stated in bit depth and chroma sub-sampling. In fact, the most important aspect in cameras' colour rendition is the colour science. It's a processing method, a complicated number of code lines that change and shift how each colour seen by the camera is represented in the final image. Each company, and each camera within each company can have a different colour rendition, something not stated in any spec sheets, making it impossible to judge how pleasing or un-pleasing the colours produced by the camera are.

 

5- Lens: In fixed lens cameras and in lens specifications of a separate lens.

The stated specs are merely the following, the focal length, the aperture values, and the size and weight of the lens. However, nothing about the lens optical performance is stated. A lens performance is a very complicated subject and it's silly one could evaluate it using the focal length and aperture values. These ''only'' state what the zoom range is, and the DOF possible. Nothing else is stated. How the resolution is, how the resolution is at different focal lengths, how the resolution is in the corners vs center, how the out-of-focus rendition looks, how much chromatic aberrations exist, colour fringing, focus breathing, Coma, Contrast, peripheral shading, nothing is mentioned. In fact, even the focal lengths mentioned are not entirely accurate on most lenses, and even the aperture values do not indicate the actual brightness (T value). Therefore a lens simply cannot be evaluated based on its spec list. The Canon 50mm F/1.4 is on specs an identical lens to the zeiss Otus 55mm F/1.4. An example, The 24-240mm F/2.8 lens on the RX10 is only better than the XC10 24-300mm F/3.5-5.6 in aperture value (DOF), yet everything else (about a dozen factors in lenses performance) about the lenses are not known and must be evaluated first, yet people on announcement crowned the 24-240mm F/2.8 the ''better'' lens ultimately, not knowing it could be a much better or a much worse lens. Specs simply don't give enough data, actually in case of lenses, almost none.

 

6- Codec, is always stated next, as H.264, ProRes, along with the bitrate and bitdepth and chroma subsampling rate.

That is accurate and it does state the compression method used in the camera, but what's important to know is that it does not indicate in anyway how the quality of the image and codec are going to be. That's determined by the implementation of the actual codec in the specific camera, and the compression quality is determined by many other elements not stated in the spec list, such as the actual resolution going into the codec, the downsampling method, the processor performance, the colour dye and colour science within the camera, and the final image enhancing algorithms. Prominent examples are a camera like the Canon C100 which has a 24mbps H.264 4:2:0 8bit codec, yet has a strikingly better compression performance than a 550D with an identical on paper 24mbps 4:2:0 8bit H.264 codec. Therefore the codec numbers on the spec sheet mean very little to almost none in the final quality of the image and even in the final compression performance, as in how hard the image can be pushed in post. The elements that do mean something are simply not stated in specification lists.

 

7- Frame rates, are always stated next

Yes they do give an accurate representation of the camera's maximum frame rate ability. But just that, all the other more important data about that mode are not stated. The downsides implemented to achieve these higher frame rates are not mentioned, for most cameras they achieve it through skipping lines, which results it horrible aliasing and a significant resolution less at higher frame rates, even on high end cinema cameras. Therefore, when the spec sheet mentions a camera can do 1080p at 25p and 50p, it's true, but whether the 50p mode is usable in quality is an entirely different subject again, not stated in the spec sheet. So it's not the frame rate number is not indicative of the camera's capability as a slow-motion shooter.

 

8- Written next can be the Audio specs.

What's mentioned is the format, hz number, the type of input/output. However, the most important elements are not stated. The type and quality of the Pre-amps in the camera, the actual sound quality, whether the audio can be controlled manually or not, therefore one simply cannot get any indication of the camera's capability to record sound based on the spec sheet, it's almost a useless section of the sheet.

 

9- Battery:

Is mentioned in the spec list as the product name, as an LP-E6n or an XD-T62 or a WTF-876 and sometimes it mentions a capacity number. However, the spec list does not mention the power draw of the camera whilst shooting video, it doesn't give any indications whatsoever on how good or bad the battery life in the camera is.

 

10- There are many very vital elements in cameras that are not mentioned altogether.

-Aliasing performance 
-Moire performance 
-Rolling shutter amount 
-Highlight roll-off 
-Resolution & quality of crop modes 
-LCD & EVF quality and performance under day light 
-The menu system speed 
-The menu system user interface 
-The responsiveness of buttons and dials 
-The feel of the camera in hand 
-The ergonomics of the handling of the camera 
-The heat management 
-The durability under various conditions 
-The materials used in the built

 

11- The elements that ARE accurate and mentioned in spec sheets are actually:

-The size of the camera 
-The size of the sensor
-The number of pixels on the total sensor
-The weight 
-The media slots
-The LCD size 
-The total number of pixels on the sensor 
-The price of the camera

That's it.

If you want to review a camera based on that be my guest it's fine, but it's just a physical impossibility to know whether this camera is ''good'' or ''bad'' when you have absolutely no idea about the 11 elements of image quality it produces, not even one including resolution.

This was the answer to the question about whether spec sheets provides enough data and the answer is a clear no.

 

Next time there's a new announcement, please think about the meaning of the represented data and whether they can be used to evaluate the camera. A camera is way more than just a sheet from the maker, that sheet only presents about 10% of the quality of the final product. Please, don't buy a camera based on specs, but it based on the quality of the camera's image to you, and the quality of the design, nothing else, and this data can only be obtained by using and testing the camera heavily, or by reading the review of someone you trust who does that.

 

Let's look at a recent spec-impressive announcement which we all crowned a wonderful camera just based on the list, The Sony A7RII. 

We're excited about the 4K resolution, but do we know how much resolution it's going to actually resolve?

in FF or s35 modes?

Do we know whether it has deadly aliasing/moire or not?

do we know if it will have deal-braking rolling shutter?

do we know if it will record more than 10 minutes without over heating?

Do we know how the low-light performance is?

Do we have any information of the colour science?

on the amount of DR?

on the Highlight roll-off?

on the noise pattern and texture?

do we know if the battery is going to last more than half an hour with a 42mp 4K sensor?

Do we know if the 5 axis stabilization will give acceptable results in video mode?

Do we know if it won't have unacceptable audio quality?

Do we know how the still images are going to look?

the compression in RAW and JPEG codecs and whether they're horrible compression methods or not?

Do we know how long it is going to be process and clear the buffer? Do we know the start-up time?

Do we know whether the camera will be broken instantly under rain? Dust? High temperature?

Do we know if the slow-motion will have SD resolution and unacceptable aliasing?

Do we know if the video AF works well?

Do we know if the codec won't have enormous banding, macro-blocking, mosquito noise, edge pixelation issues?

Do we know anything about the camera?

We don't,

yet 99% of the PR readers came out believing firmly it's a great camera, without having any data to support it.

(The A7RII might do extremely well in ALL these areas but the point is made I believe)

 

Specs only mean the size and weight of the camera. Wait, in fact, the listed size and weight are always misleading as it's stated without a battery, cards, handles, lens, microphone, oh well.

Just please stop reviewing camera on spec-sheets, it's only harming the film-making community and leading people to choose incorrect products. 

What a kickass post. Mr. Sadaawi most things concerning most of the relevant points youbhave raised are only judged once the camera is in the hands of the users. Most manufacturers have absolutely no idea about even basic features any they must be there.

I read a lot of reviews of the blackmagic cameras saying that they actually have about 8 stops of dynamic range. The reason they still look so good is because they shoot in ProRes and Raw and hence you can push almost everything to look better. 

Manufacturers can test battery life but when certifying agencies do a more standard testing test why should manufacturers try and give unstandardised figures. 

The only way cameras can be tested better, is for early testers to be given cameras atleast a few weeks before release (like in the case of the G7), and let unbiased users give you feedback instead of DoP who may have a monetary obligation to only say the nice things. 

[TheRenaissanceMan]

I read a lot of reviews of the blackmagic cameras saying that they actually have about 8 stops of dynamic range. The reason they still look so good is because they shoot in ProRes and Raw and hence you can push almost everything to look better. 

​Completely untrue. Low DR footage doesn't gain more DR from being encoded in a better format. It just becomes easier to shift that DR around. The Blackmagic cameras have between 12 and 13 usable stops, depending how you define usable. 

[Guest Ebrahim Saadawi]

@Sanveer Agree with every word, Well said (except for the BM part which @TheRenaicassanceMan addressed correctly) As you said, the only point of writing this long post is to convince people it's impossible to know whether a camera is absolute doh shit or God greatness personified in a camera, both of these cameras in fact have an identical spec sheet The only, only way it to test it, by someone who has great experience in the field of testing cameras. Actually, no to be offensive but I still find I have seen no single reviewer/person that has the ability to give a 100% complete technical review of a camera. I wish a reviewer, someone the ability to receive new products, would build a testing facility for video cameras and test comparing cameras based on ALL their technical aspects that differentiate cameras (as opposed to taking the camera out on a day shoot and say it looks good/bad and address a couple of elements which all reviewers do, all of them) Imagine we'd have a comprehensive database we can go through and see how a camera performs in each field with a click. Want to see how the GH4 resolution compares to the NX1, have both files presented side by side in a completely identical setting, scientifically, want to know how the A7s DR compares to the C100, get two identical images and push/pull and see exactly that, same with all the other 11 elements of high image quality. That would be the greatest gift to the filmmaking community. 

[sanveer]

​Completely untrue. Low DR footage doesn't gain more DR from being encoded in a better format. It just becomes easier to shift that DR around. The Blackmagic cameras have between 12 and 13 usable stops, depending how you define usable. 

I meant the codec allows for pushing grading really well, and not some magical appearence of more stops of dynamic range because of codec. I never said something even suggesting that.

 

@Sanveer Agree with every word, Well said (except for the BM part which @TheRenaicassanceMan addressed correctly) As you said, the only point of writing this long post is to convince people it's impossible to know whether a camera is absolute doh shit or God greatness personified in a camera, both of these cameras in fact have an identical spec sheet The only, only way it to test it, by someone who has great experience in the field of testing cameras. Actually, no to be offensive but I still find I have seen no single reviewer/person that has the ability to give a 100% complete technical review of a camera. I wish a reviewer, someone the ability to receive new products, would build a testing facility for video cameras and test comparing cameras based on ALL their technical aspects that differentiate cameras (as opposed to taking the camera out on a day shoot and say it looks good/bad and address a couple of elements which all reviewers do, all of them) Imagine we'd have a comprehensive database we can go through and see how a camera performs in each field with a click. Want to see how the GH4 resolution compares to the NX1, have both files presented side by side in a completely identical setting, scientifically, want to know how the A7s DR compares to the C100, get two identical images and push/pull and see exactly that, same with all the other 11 elements of high image quality. That would be the greatest gift to the filmmaking community. 

I Conpletely agree. Most camera reviewers are so Off everything that matters. Also, 2 reviewers will be off figures by such a large sum, that it become suspicion arousing for readers. Also, a lack of balance in reviewers language can make one assume too much. And, finally the Cult of Cameras, makes some cameras seem absolutely infallible.

I also wish stores All Over the World, as opposed to only the US and Canada had some sort of return policies. I am from India, and apart from the fact that a lot of cameras never make it, there is absolutely no testing and return policy. Also, the taxes on cameras are second only to UK. 

 

[MattH]

I never wrote an article on which everyone agrees, that's great, and even well, boring if I may  

anyone? 

specs don't mean absolutely nothing, so let's discuss that, what do they actually give?

​Hope.

For example, I hear that the new NX mini will have 4k video.  That gives me hope that instead of the shitty 1080 video (which I would have known for sure had it been 1080 in the specs) it may have good 4k video.  But I am prepared for it being shitty 4k video as well.  One thing I know from the spec sheet is that it will have a 20mp sensor.  From this I know that it will either line skip etc giving artifacts, or do a full sensor readout which will probably be slow and give bad rolling shutter.  But I still have hope.  Could provide a truely poketable 4k option.  I have less hope for the RX100 iv after seeing the awful sample on youtube, but still a bit of hope that the shooter and editor fucked up.

[TheRenaissanceMan]

I find specs are most useful in indicating the intent of the camera. When we saw the NX1's specs--4K video, H.265, high frame rates, manual control--it showed us that Samsung had put serious work into making a competitive hybrid camera. When we saw the Blackmagic Pocket's specs--no frills, RAW and ProRes recording, S16 sensor size, small form factor--we knew they wanted to make a small, affordable cinema camera. Marketing ties into this; Blackmagic wouldn't tout "13 stops of dynamic range" and "the look of real digital film" if they were making a consumer camera. 

[sanveer]

@Sanveer Agree with every word, Well said (except for the BM part which @TheRenaicassanceMan addressed correctly) As you said, the only point of writing this long post is to convince people it's impossible to know whether a camera is absolute doh shit or God greatness personified in a camera, both of these cameras in fact have an identical spec sheet The only, only way it to test it, by someone who has great experience in the field of testing cameras. Actually, no to be offensive but I still find I have seen no single reviewer/person that has the ability to give a 100% complete technical review of a camera. I wish a reviewer, someone the ability to receive new products, would build a testing facility for video cameras and test comparing cameras based on ALL their technical aspects that differentiate cameras (as opposed to taking the camera out on a day shoot and say it looks good/bad and address a couple of elements which all reviewers do, all of them) Imagine we'd have a comprehensive database we can go through and see how a camera performs in each field with a click. Want to see how the GH4 resolution compares to the NX1, have both files presented side by side in a completely identical setting, scientifically, want to know how the A7s DR compares to the C100, get two identical images and push/pull and see exactly that, same with all the other 11 elements of high image quality. That would be the greatest gift to the filmmaking community. 

​

I find specs are most useful in indicating the intent of the camera. When we saw the NX1's specs--4K video, H.265, high frame rates, manual control--it showed us that Samsung had put serious work into making a competitive hybrid camera. When we saw the Blackmagic Pocket's specs--no frills, RAW and ProRes recording, S16 sensor size, small form factor--we knew they wanted to make a small, affordable cinema camera. Marketing ties into this; Blackmagic wouldn't tout "13 stops of dynamic range" and "the look of real digital film" if they were making a consumer camera. 

​

I kind of feel I agree with quite a few things, and have a few extra things to say.

Like the RenaissanceMan (???) said above, the intention of the company is clearly outlined by the spec sheets. Also, I believe that  while engineers may be great at technology, they may be quite stupid where user experience and requirements are concerned, among many other things. That is why, good companies always give their engineers as much user feedback as possible, and Sony and Panasonic have been listening to users, more and more, of late. So, while for almost every conceivable reason understandable, it becomes impossible for Sony and Panasonic to give you the quality of an Alexa, for under $1000, they can give you pretty good high end consumer cameras with very small footprints, like the Rx100 IV and the LX100.

Also, I believe, that specs and firmware, are also inter-related, and while specs for future products are based on user reviews and feedback, they are also related to user dictated firmware updates. Though, I wish, companies would also address issues with regard to hardware, like the lens coating on the LX100, or something else that can easily be handled without noticeable changes to the overall body.

I wish Panasonic would add Log to the LX100, even if it were a paid one. That would help it compete better with the RX100 IV, slow motion being the only other advantage (and the LX100 having many advantages over the RX100 IV). 

You know, the last time they tried to be, ahem, too scientific about camera testing, they had that famous Zacuto Shootout, where the GH2 beats the pants off of most other cameras, in their blind testing (or so reported ... :P). I am guessing, that, for reasons like that one, they keep only cameras within similar price points, to compete with each other.