Canon EOS R5 so-called overheat timer defeated by a single screw in battery door

[docmoore]

14 minutes ago, horshack said:

Thanks. That probably saves me from buying ISObuster for this effort. Was the size of the file about what it should've been for how long the video was? If so then the file may have all the salvageable data recovered and just needs to have the tail of the MP4 container repaired as discussed earlier.

 I haven't been getting the .DAT but just realized it's because I'm using FAT32 whereas everyone else is likely using exFAT. It defaulted to FAT32 due to the smaller card I was using (32GB), which I chose to make the imaging/recovery experiments faster. If I switch the same card manually to exFAT then the .DAT file starts showing up after an interrupted recording on my Canon RP. I'll be focusing on exFAT in my ongoing work since that's what most are probably using.

Very short recording ... only 1.38GB but that appears about right for the 5.5K RAW it was recording. I believe you are correct ... the question is how to add the container and whether the EXIF data can be written.

[horshack]

11 minutes ago, docmoore said:

Very short recording ... only 1.38GB but that appears about right for the 5.5K RAW it was recording. I believe you are correct ... the question is how to add the container and whether the EXIF data can be written.

Good. The EXIF can always be grafted with exiftool on but it's probably superfluous for desktop NLE use. Can you possibly share the file via Dropbox or Google Drive? Could you also take another video of the same length that's recorded normally and share that with me as well? I'll be generating my own files too but the process on wiping the cards to perform each experiment is slowing me down so I have free time in between each.

[docmoore]

Easily done but would you rather have a file that is not Canon Raw ... their Canon Raw Developer and Davinci Resolve ... I assume FCP ... are needed ....

I can record clips in 4K DCI all-I @ 23.976 if that is better for you.

[horshack]

3 minutes ago, docmoore said:

Easily done but would you rather have a file that is not Canon Raw ... their Canon Raw Developer and Davinci Resolve ... I assume FCP ... are needed ....

I can record clips in 4K DCI all-I @ 23.976 if that is better for you.

Oops, forgot you mentioned raw. I definitely want to focus on processed first. Can you shoot a pair of H.264 4K 24P videos? I'll work on H.264 first then consider H.265 and/or ALL-I

[wolf33d]

3 hours ago, Andrew Reid said:

The other ugly thing is that I have taken an absolute beating online with people questioning my reputation left right and centre for revealing this timer and behaviour from Canon.

Canon meanwhile, brand reputation virtually untouched, not two hoots from them, no statement, no recall and the EOS R5 continues to fly off shelves!!

BBC and CNN tech reporters should be all over it. Where are they?

I have been insulted also on other forums, when I explained why I strongly believed it was a Canon limitation on purpose (once we got various bits of info like the camera keeping same behavior regardless of temperatures etc) on the early days. 
Those people / other sites shut their mouth now. But who cares...
I and a lot of other people appreciate the digging you have done.

Don’t worry about Canon their image is impacted you can be sure of that. Tons of people around me are just like me maintaining their preorder because they have a chance to make it right but if they don’t they will go fuck themselves with their camera. 
A lot of people at Canon need to be fired. It’s a big mess. They just have to make it right... but those Japanese you know what it means for them loosing the face....

All they need to do is a small SW update to make it the best hybrid on the market. If they don’t fix it that means somewhere between 80-99% of hybrid shooters (= who care about video) will go elsewhere. I won’t buy their EOS 70 cine mini camera. It won’t be good in photo. Give us the god damn hybrid photo video tool we need. 

[docmoore]

15 minutes ago, horshack said:

Oops, forgot you mentioned raw. I definitely want to focus on processed first. Can you shoot a pair of H.264 4K 24P videos? I'll work on H.264 first then consider H.265 and/or ALL-I

Check messages ...

[docmoore]

It was completely avoidable ... those EOS Cine guys must be the alpha dogs ....

I assume there is a good reason for the limits ... and that they chose the easy quick path to get the camera to market. You want video ... sure ...

and when it hurts bad enough join us in the big leagues. If they had included a cooling system then they have no excuse to protect the EOS Cinema line.

[herein2020]

4 hours ago, Andrew Reid said:

The other ugly thing is that I have taken an absolute beating online with people questioning my reputation left right and centre for revealing this timer and behaviour from Canon.

Canon meanwhile, brand reputation virtually untouched, not two hoots from them, no statement, no recall and the EOS R5 continues to fly off shelves!!

BBC and CNN tech reporters should be all over it. Where are they?

  

38 minutes ago, wolf33d said:

I have been insulted also on other forums, when I explained why I strongly believed it was a Canon limitation on purpose (once we got various bits of info like the camera keeping same behavior regardless of temperatures etc) on the early days. 
Those people / other sites shut their mouth now. But who cares...
I and a lot of other people appreciate the digging you have done.

Don’t worry about Canon their image is impacted you can be sure of that. Tons of people around me are just like me maintaining their preorder because they have a chance to make it right but if they don’t they will go fuck themselves with their camera. 
A lot of people at Canon need to be fired. It’s a big mess. They just have to make it right... but those Japanese you know what it means for them loosing the face....

All they need to do is a small SW update to make it the best hybrid on the market. If they don’t fix it that means somewhere between 80-99% of hybrid shooters (= who care about video) will go elsewhere. I won’t buy their EOS 70 cine mini camera. It won’t be good in photo. Give us the god damn hybrid photo video tool we need. 

I experienced a bit of that myself and I only said a fraction of what Andrew and others have put out. I was banned from Canon Rumors because I didn't give Canon a pass like everyone else and seemed to be the first one on that forum that said the R5 and R6 were not fit for purpose if the overheating was a real thing in the production cameras (this was when everyone was still saying it is a pre-production issue).

For me it worked out perfectly though..I don't want to be a member of a forum where you are not allowed to speak the truth about a product without being banned. I was also banned from the DJI forums because there is a critical issue that affects it's operation that DJI refuses to acknowledge or fix and I seemed to be the only one continuing to ask them for an update (yrs later it still is not fixed).

[horshack]

42 minutes ago, docmoore said:

Check messages ...

Thanks for the files. Tried two separate automated MP4 repair programs/scripts, both of which support reading the headers/metdata from the "good" file and attempt to graft that data over those missing elements in the bad file, specifically the missing moov atom in the corrupt file. No joy so far. Oddly when I examine the "good" file the moov atom is actually at the beginning of the file - I expected it to be at the end since that's what's missing from the corrupt file. It might be that Canon relocates the moov atom to the front after a successful recording end. It's usually placed at the beginning of files to facilitate better streaming as the file is being downloaded. Or it might be that IsoBuster didn't recover the file correctly. I'll keep digging.

Here's the Atom data from the good file, obtained from AtomicParsley <filename> -T

Atom ftyp @ 0 of size: 28, ends @ 28
Atom moov @ 28 of size: 262116, ends @ 262144
     Atom uuid=85c0b687-820f-11e0-8111-f4ce462b6a48 @ 36 of size: 65750, ends @ 65786
     Atom udta @ 65786 of size: 2056, ends @ 67842
         Atom manu @ 65794 of size: 20, ends @ 65814                     ~
         Atom modl @ 65814 of size: 38, ends @ 65852                     ~
         Atom urat @ 65852 of size: 16, ends @ 65868                     ~
         Atom free @ 65868 of size: 1974, ends @ 67842
     Atom mvhd @ 67842 of size: 108, ends @ 67950
     Atom trak @ 67950 of size: 16328, ends @ 84278
         Atom tkhd @ 67958 of size: 92, ends @ 68050
         Atom edts @ 68050 of size: 36, ends @ 68086
             Atom elst @ 68058 of size: 28, ends @ 68086
         Atom mdia @ 68086 of size: 16192, ends @ 84278
             Atom mdhd @ 68094 of size: 32, ends @ 68126
             Atom hdlr @ 68126 of size: 33, ends @ 68159
             Atom minf @ 68159 of size: 16119, ends @ 84278
                 Atom vmhd @ 68167 of size: 20, ends @ 68187
                 Atom dinf @ 68187 of size: 36, ends @ 68223
                     Atom dref @ 68195 of size: 28, ends @ 68223
                 Atom stbl @ 68223 of size: 16055, ends @ 84278
                     Atom stsd @ 68231 of size: 311, ends @ 68542
                         Atom hvc1 @ 68247 of size: 295, ends @ 68542             ~
                     Atom stts @ 68542 of size: 24, ends @ 68566
                     Atom stss @ 68566 of size: 296, ends @ 68862
                     Atom ctts @ 68862 of size: 4472, ends @ 73334
                     Atom stsc @ 73334 of size: 28, ends @ 73362
                     Atom stsz @ 73362 of size: 3364, ends @ 76726
                     Atom co64 @ 76726 of size: 6704, ends @ 83430
                     Atom sdtp @ 83430 of size: 848, ends @ 84278
     Atom trak @ 84278 of size: 8276, ends @ 92554
         Atom tkhd @ 84286 of size: 92, ends @ 84378
         Atom edts @ 84378 of size: 36, ends @ 84414
             Atom elst @ 84386 of size: 28, ends @ 84414
         Atom mdia @ 84414 of size: 8140, ends @ 92554
             Atom mdhd @ 84422 of size: 32, ends @ 84454
             Atom hdlr @ 84454 of size: 33, ends @ 84487
             Atom minf @ 84487 of size: 8067, ends @ 92554
                 Atom smhd @ 84495 of size: 16, ends @ 84511
                 Atom dinf @ 84511 of size: 36, ends @ 84547
                     Atom dref @ 84519 of size: 28, ends @ 84547
                 Atom stbl @ 84547 of size: 8007, ends @ 92554
                     Atom stsd @ 84555 of size: 103, ends @ 84658
                         Atom mp4a @ 84571 of size: 87, ends @ 84658
                             Atom esds @ 84607 of size: 51, ends @ 84658
                     Atom stts @ 84658 of size: 24, ends @ 84682
                     Atom stsc @ 84682 of size: 748, ends @ 85430
                     Atom stsz @ 85430 of size: 6548, ends @ 91978
                     Atom co64 @ 91978 of size: 576, ends @ 92554
     Atom trak @ 92554 of size: 947, ends @ 93501
         Atom tkhd @ 92562 of size: 92, ends @ 92654
         Atom mdia @ 92654 of size: 847, ends @ 93501
             Atom mdhd @ 92662 of size: 32, ends @ 92694
             Atom hdlr @ 92694 of size: 33, ends @ 92727
             Atom minf @ 92727 of size: 774, ends @ 93501
                 Atom nmhd @ 92735 of size: 12, ends @ 92747
                 Atom dinf @ 92747 of size: 36, ends @ 92783
                     Atom dref @ 92755 of size: 28, ends @ 92783
                 Atom stbl @ 92783 of size: 718, ends @ 93501
                     Atom stsd @ 92791 of size: 50, ends @ 92841
                         Atom tmcd @ 92807 of size: 34, ends @ 92841
                     Atom stts @ 92841 of size: 24, ends @ 92865
                     Atom stsc @ 92865 of size: 40, ends @ 92905
                     Atom stsz @ 92905 of size: 20, ends @ 92925
                     Atom co64 @ 92925 of size: 576, ends @ 93501
     Atom free @ 93501 of size: 168643, ends @ 262144
Atom mdat @ 262144 of size: 730194240 (^), ends @ 0
             (^)denotes a 64-bit atom length

 ~ denotes an unknown atom
------------------------------------------------------
Total size: 730456384 bytes; 66 atoms total. AtomicParsley version: 0.9.0 (utf16)
Media data: 730194240 bytes; 262144 bytes all other atoms (0.036% atom overhead).
Total free atom space: 170617 bytes; 0.023% waste. Padding available: 1974 bytes.
------------------------------------------------------

[docmoore]

I did not use IsoBuster for these ... just copied the .dat file to the desktop.

Other than structure ... since both files are unique ... with regards to time of filming and microstucture of the capture elements ... it may be of use

only in finding the commonalities that exist in them.

Not for the feint of heart ... reverse engineering without a manual ...

This engages me more than the camera specifics ... too crippled to warrant purchase.

[gethin]

5 hours ago, Andrew Reid said:

The other ugly thing is that I have taken an absolute beating online with people questioning my reputation left right and centre for revealing this timer and behaviour from Canon.

Canon meanwhile, brand reputation virtually untouched, not two hoots from them, no statement, no recall and the EOS R5 continues to fly off shelves!!

BBC and CNN tech reporters should be all over it. Where are they?

In Australia a federal mp funnelled millions of dollars of tax payers money into an offshore company via selling water rights for a piece of land that gets no water.  Mainstream media: tumbleweeds.

The only way this will hit the news will be if either someone sues cannon or badgers a consumer rights department enough to do something.  Also Choice might have a look. 

 

 

 

[Andrew Reid]

Magically Screwed!

 

2 hours ago, horshack said:

Thanks for the files. Tried two separate automated MP4 repair programs/scripts, both of which support reading the headers/metdata from the "good" file and attempt to graft that data over those missing elements in the bad file, specifically the missing moov atom in the corrupt file. No joy so far. Oddly when I examine the "good" file the moov atom is actually at the beginning of the file - I expected it to be at the end since that's what's missing from the corrupt file. It might be that Canon relocates the moov atom to the front after a successful recording end. It's usually placed at the beginning of files to facilitate better streaming as the file is being downloaded. Or it might be that IsoBuster didn't recover the file correctly. I'll keep digging.
------------------------------------------------------

Thanks for your good work Horshack!

I'll be on hand again tomorrow if you need me try anything at this end.

[elgabogomez]

So, the r5 is a [ For the professional image-maker who needs resolution, speed, and video capabilities, there is the Canon EOS R5. Setting a new standard for versatility, this full-frame mirrorless camera features a newly developed 45MP CMOS sensor, which offers 8K raw video recording, 12 fps continuous shooting with a mechanical shutter, and is the first EOS camera to feature 5-axis sensor-shift image stabilization.]* (b&h description) product that can record 16 minutes chunks of 8k footage (hard power off/power on required each new chunk) without overheating... as long as you don’t care about using those recordings. 
So is it a proof of concept device that you can buy to support and thank canon for past, present and future innovations??

[Sicofante]

The way I see this fiasco is optimistically.

First of all, Canon engineers made an astounding camera. They made a 4.000 $ mirrorless body camera able to record 8K RAW indefinitely without a hiccup. Then, of course, the marketing dept. saw this and, terrified of having their cine cameras going into the dustbin, required the engineers to make something about it. Some "genius" came out with the stupid idea of both fake overheating and the fake cool down period (which smelled fishy to anyone building computers or knowing just a little bit about electronics).

But now we know an 8K RAW camera with unlimited recording can be done at the size of a mirrorless body. We know and Canon competitors know too. Sony or Panasonic have cine cameras to protect, but Fuji, Nikon or Sigma don't. I bet engineers in these companies are working, as we speak, on making something similar to what Canon did. And they will succeed, eventually.

We are in for a big jump in video recording from small cameras. Thank you Canon!!!

[Andrew Reid]

It has yet to be proven that 8K RAW goes indefinitely, but yes it is very impressive hardware.

There may be thermal throttling or lock-ups somewhere during long sessions, we don't know - but what is now proven without a shadow of a doubt is how artificial the 15 minute limit is and how it counts down on a timer rather than a high maximum temperature cut off point.

My opinion gets stronger every day that all this is about segmenting it from the Cinema EOS cameras.

But we all know this is a stupid business decision, because most Cinema EOS camera owners wanted to buy an EOS R5 as a b-cam, second body, third body, handheld rig, you name it.

In my opinion Canon seems to want to sell every professional shooter a minimum of 2 very expensive cameras for the same shoot - one single hybrid camera at $4000 plus very expensive high margin optics is not enough it seems. Shoot stills? Buy EOS R5. Shoot video? Buy C500 Mark II. Shoot both. Buy both.

Yup it is terrible for us who just want to buy the EOS R5 for it's amazing hardware and cinematic images.

We're always the one to get screwed it seems.

This is why Canon, in my opinion, has been losing video users for years.

It is faulty thinking, throughout the company. I don't know which part of Canon is most to blame - USA or Japan? Both? EOS or Cinema? Maybe both? Or do the EOS guys secretly really dislike the politics that the Cinema business has brought to the company and the restrictions it has placed on their products and all the stupid segmentation?

Either way, they must change.

I am up to 1 hour of 8K H.265 which is very special from a 45MP full frame sensor.

It makes it even more frustrating that Canon has turned it into a lemon then!

If Canon chose to lie to their customers about limitations on a $700 consumer camera that was boring and unexciting, the passion wouldn't quite be as strong to uncover the truth and fix it... That it is a very expensive flagship camera and the first of its kind in the world, makes it a hugely emotional topic for me & you, and rightly so.

[Emanuel]

Their point is the number of people who will simply NOT buy anything else for higher.

This camera can be as revolutionary as the perfect hybrid would be if/once these crippling limits will be lifted.

 

Now remains to see how overheating can be a real issue.

 

There are borders, of course. But enough to limit it as they did through firmware?!!

C'mon... no fools over here ;- )

[ajay]

The Canon rumor sites are buzzing about two new cinema cameras with the R-mount. That's probably why this camera has been crippled.

What Canon should have done is marketed the R5 as a high-end stills camera for $3.5k with mediocre 4k modes and for a $500 upgrade that would include high-end video options. I think that would have gone over a lot better than the mess they have made. Lots of ill-will with Canon right now.

[Emanuel]

$500 upgrade premium is too short for their greed... Sad but true.

[horshack]

9 hours ago, docmoore said:

Check messages ...

Made a lot of progress on understanding the .DAT files. Using my Canon RP as a test bed, I recorded two 1:00 videos, one regularly and the other power-interrupted. I then copied the entire SD card for each case as a file image to my system for analysis. Prior to recording video each SD was prepared by fully wiping it with dd, then formatting in the camera. I also went back to FAT32 because it's easier to analyze those structures than exFAT.

This yields two very similar filesystem images. For the regularly-recorded video SD image I see a single data file (MVI_0478.MP4 in this specific case). However if I analyze the directory entries in the FAT32 structure I see a second file that's been deleted, named ?VI_047.DAT. The '?' is there because the file has been deleted (0xE5 is the FAT32 marker to indicate a deleted directory entry), meaning its original name is MVI_0478.DAT. Even though MVI_0478.DAT has been deleted its FAT32 cluster location and most-recent file size is still available in the directory entry.

Looking at the contents of the .DAT I see an "mdat" field. This is the raw MP4 data holding the video/data of the video - it's called an "mdat atom" in MP4 container parlance. The size of the .DAT file is exactly 128KB less than the size of the .MP4. What's more, the on-SD location of the .DAT is exactly 128KB after the .MP4, which means the two files are actually contiguous with each other.

So the camera wrote the .DAT first (while recording the video) but left 128KB before it pre-allocated in the FAT32 cluster table. When the video recording stopped it then wrote the 128KB before the .DAT, which contains all the other MP4 atoms necessary to represent the file, including the 'moov atom' which was reported missing when trying to recover the .DAT as .MP4 earlier using open-source recovery tools (failed because it's missing the full 128KB .MP4 header).

The 128KB MP4 container header stuff + mdat atom content represents the full MP4 file.

So the .DAT files we're seeing for the interrupted recordings on the R5 are the actual raw video/audio data sans the proper 128KB MP4 header area.

On my Canon RP the .DAT directory entry is missing for the interrupt recording. However if I go to the location on the interrupted-SD image as where the .DAT file exists on the good-SD image I see the mdat data. So it appears that on my RP the camera writes to this area during recording without first setting the directory entry to reflect that area is allocated. If you're able to see the .DAT files from your 1DX that means the behavior of that camera is different. Not sure how the R5 acts. Either way, recovering the .DAT file for my RP is trivial because I know what cluster it begins on by comparing it to the good-SD image.

I found the .DAT area on the interrupted-SD image and spliced in the 128KB MP4 header from the .MP4 from the good image. The file plays...and I hear a quick snippet of sound, but no video. So there is work to do there. But at least it demonstrates the basic integrity of the full MP4 is in place.

To get this working on R5 images we'd have to determine how to build the proper 128KB MP4 container header, the same as how the camera builds it when the user ends a video recording. This might be as simple as splicing from a "good" .MP4 image, but probably requires a lot more effort, and then graft that 128KB header data with the the .DAT file to reconstruct the full .MP4

[phoberus]

is this the same with the R6?