horshack

What's the highest EXIF temp you saw shooting stills for that sequence you noticed it slowing down? If we are seeing such throttling even shooting stills then my theory that the thermal management is to protect the cards / data rates becomes much more plausible. A continuous stills burst slowing down is manageable - a video dropping frames would be more problematic for users - which may explain why the R5 continues to allow stills but not video in hotter situations.

Yep exactly, it's to get the final temperature, although with the 8K clips chopping up at only 47.5 seconds it's probably not needed. It would be more useful for the other long-run tests though, like 4K HQ. I haven't established whether the EXIF temperature in the video files is sampled at the start or end of the recording - I would presume it's at the end when all the EXIF is built but I don't know for sure. I'm going to do an experiment on my RP today to determine this.

The slowdown may have been related to internal garbage collection rather than thermals - I would be surprised if a CFE card throttled shooting just stills. But again it's something to keep an eye on. Just sent you a PM about getting the .CSV data in text form.

Keep in mind there are other components inside the R5 that may have thermal envelopes below DIGIC and Image Sensor, the most obvious of which is the media cards. ProGrade list the maximum ambient operating temperature of their cards at 70C, beyond which they can start to throttle. If you look at the photos of the Chinese R5 teardown the CFE/SD bay sits right next to DIGIC and SDRAM. https://progradedigital.com/wp-content/uploads/2019/03/ProGrade_DS_CFexpress_Final_E.pdf

Thanks. Things are getting pretty toasty at around 72C 😀 That also happens to be the temp that Hoka Key's test stopped after 69 minutes of 8K recording for the overnight-fridge test. Here's my graph of that test (on right - left is non-fridge test):

Good, thanks. Dropping power to NAND devices while writing isn't good for their health, esp low-end consumer devices like SD cards that may not have good power-loss recovery mechanisms. They may interpret the CRC errors from those blocks on subsequent accesses as bad blocks and start to remap them, which can affect performance. To help avoid this I would try to time the battery pulls to occur in between card writes by monitoring the card-access light on the camera, presuming there are idle periods visible in between writes for the data rate of the video quality you're shooting.

Great, thanks. Please keep an eye out for those "slow card" error messages. One of theories about Canon's thermal management is to avoid overheating the media cards and causing them to throttle, esp. CFE but perhaps also SD. Once the cards get above a certain temp they start to internally throttle their performance to reduce heat, and this may manifest as those "slow card" error messages.

Thanks. Here's the batch file I wrote that will extract the EXIF temps and put them into a .CSV file. Pass it the directory where you put all the .MP4s and the final still-image, plus a *.* mask. For example: get_exif.cmd c:\pics\*.* @echo off REM Put results.csv in same directory as files we're processing SET resultsfile="%~dp1results.csv" echo %resultsfile% exiftool.exe -q -m -ext MP4 -ext JPG -ext CRW -ext CR2 -ext CR3 -p "${CreateDate;$_=(split(' ',$_))[1]},${FileName},${Duration#},${CameraTemperature#}" %1 > %resultsfile% Above a certain filesize exiftool complains about not having large file support. Hopefully that wont be the case for the split-4GB files.

Oops, that 162 MB/s is for 8K ALL-I. Using IBP it should be about 81 MB/s

Fantastic, that's a working solution for SD then. Canon might have removed the legacy FAT32 support starting with CFE since it's an easy technology inflection point to do so. Could I ask a big favor? Could you possibly run 8K recording for an hour, using battery pulls as necessary? Also, after you reach the hour could you immediate take a photo? I'll post a batch file that will extract the EXIF temperatures from all the files and put them into a .CSV file I can graph with Excel. Typing on my phone right now so don't have access to my computer this moment. Thanks again! The data rate for 8K is about 162 MB/s so you may need to copy the files halfway through so you don't run out of card space.

Thanks! Did you verify that the thermal timer didn't reset after pulling the battery for this FAT32 case? Interesting how the CFE didn't work. Hoka Key on the dpreview forums wasn't able to get his 320GB CFE card recognized by his R5 either when formatted to FAT32. I wonder if Canon just disabled FAT32 support for CFE. Do you have a smaller CFE card to try by any chance?

The goal isn't to save the parameters...the goal is to *not* save them 😀 It's the entire basis of the workaround for the timer element of the R5's thermal algorithm. Btw, you don't have to turn off the camera to save the parameters. Taking a photo or a video or moving the mode dial between stills/movie mode saves them as well.

@Jn- came up with an alternate idea - use FAT32 on the card and the camera will automatically split up the files into 4GB. I don't have an R5 but I tested this on my RP and verified it doesn't save the NVRAM settings for each of the 4GB file splits. You can read more about it in the following two links. I've asked Andrew and J. Marcus to try this on their R5 but I don't think they've had a chance yet. Prior to that I did some work on recovering the .DAT file but stopped after the above solution looked like a better alternative. You can read about the work I did in the following link:

Update: I recorded 25:00 of video then battery pull with no issue on my RP. All ~4GB MP4 segments were fine, with naturally the last one missing Don't let the recording reach the 29:59 legacy limit - if you do the camera will save the NVRAM settings when it automatically ends the recording The Ridgecrop formatter tool link I posted for Windows only worked once for me. After that the camera wouldn't accept any cards formatted with it. I then tried the formatter built into OSX using the FAT32 and MBR options (not GPT) - the camera accepts those cards but then displays an error whenever you take a photo or end a video recording. The only reliable and consistent way I've found to format > 32GB cards for FAT32 that my RP would accept and work with is under Linux. You can see instructions how to do so here. Here's a screenshot of the ~4GB segements on a 64GB SD card formatted to FAT32 for my 25:00 recording:

Just tried it for a longer video, this time 10:00, to test two splits and it still works. Two perfectly-good .MP4 files, each approx 4GB and 4:41 in length, and an RP with amnesia after powering it back on. The only limitation of this technique is 32GB, since Canon cameras format all cards > 32GB as exFAT, even though FAT32 supports volumes up to 2TB. But there's a workaround for that as well - format the >32GB card on computer instead. Windows 7/10 wont let you format cards > 32GB to FAT32 but there are third-party utilities that will. I just used http://www.ridgecrop.demon.co.uk/index.htm?guiformat.htm on a 64GB card and the camera accepted the card with no complaints. Recording to it right now on my RP to test splits for a very long video. And while Windows wont let you format >32GB cards to FAT32 it will mount them just fine.

You sir win the internet today. I just tried it on my Canon RP, which like the R5 forgets any changes to the camera's settings (like exposure settings) if you pull the battery in the middle of the recording but will remember the changed settings if you stop a recording normally. Procedure: Tape down the door sensor and leave the door open so the battery is accessible Put a 32GB card in the camera and Turn the camera on and format the card. It will automatically use FAT32 cards with a capacity <= 32GB Start a video recording, in this case I used 4K 24P, which has an average total video/audio bitrate of 14.43 MB/s. Shoot a video whose length will exceed 4GB. For my 4K 24P that's any video longer than just under 5:00. I shot one for exactly 5:00 Pull the battery Check the card and verify there's a single .MP4 file Reinsert the battery and verify the camera "forgot" the previous session's settings We know the camera saves its settings whenever it has successfully recorded a video. The question is whether @Jn- idea will avoid that save for the FAT32-case of splitting up a video file at 4GB/each (FAT32 limit), provided we still pull the battery at the end of the recording. In other words, will the camera save NVRAM for each intermediate 4GB-split file or only for the final file after properly stopping a recording? Drumroll...It works! My RP's aperture was f/4 when I turned it on for step #3. I changed it to f/11 for the test. After reinserting the battery my RP's aperture reverted back to f/4! Andrew, please test this ASAP on your R5 and see if it matches my RP's behavior of forgetting the settings, the most important of which is of course the thermal timer.

Matt Granger posted a video yesterday:

All-I for the pair of videos. thanks. I think 30 seconds might be too big to share - 15 seconds/each should be plenty.

I'm working with 4K H.264 files from a 1DX that @docmoore provided me. I can actually use a pair of R5 ALL-I 4K files - perhaps those will be easier to reconstruct. Can you shoot two 4K ALL-I videos for me, each 30 seconds in length, one that is recorded normally and the other where you pull the battery after 30 seconds. The first will be a regular .MP4 and the second will be a .DAT file. Then put them up on a share I can download from? Please have some kind of sound playing while recording - that will help me in the debugging effort find and sync up video/audio frames. Thanks!

I think I've found out why the video playback is blank and the there's no audio. The MP4 header has 'stsz' and 'stco' atoms that are indexes into the actual video and audio data in the mdat atom. Those atoms are used by players/NLEs to correlate time to video/audio frames. Since I copied the entire MP4 header from the "good" video those indexes will be wrong for the "bad" video and have to be rebuilt. I found an open-source tool named uncut that does this (https://github.com/ponchio/untrunc), documentation here. Built and ran it but it wasn't able to find any of the frames in the files. Will be looking into the code to see if it can be tweaked to work on these files

Petapixel has picked it up: https://petapixel.com/2020/08/24/simple-hack-proves-canon-eos-r5-overheating-limit-is-artificial/

Made a lot of progress on understanding the .DAT files. Using my Canon RP as a test bed, I recorded two 1:00 videos, one regularly and the other power-interrupted. I then copied the entire SD card for each case as a file image to my system for analysis. Prior to recording video each SD was prepared by fully wiping it with dd, then formatting in the camera. I also went back to FAT32 because it's easier to analyze those structures than exFAT. This yields two very similar filesystem images. For the regularly-recorded video SD image I see a single data file (MVI_0478.MP4 in this specific case). However if I analyze the directory entries in the FAT32 structure I see a second file that's been deleted, named ?VI_047.DAT. The '?' is there because the file has been deleted (0xE5 is the FAT32 marker to indicate a deleted directory entry), meaning its original name is MVI_0478.DAT. Even though MVI_0478.DAT has been deleted its FAT32 cluster location and most-recent file size is still available in the directory entry. Looking at the contents of the .DAT I see an "mdat" field. This is the raw MP4 data holding the video/data of the video - it's called an "mdat atom" in MP4 container parlance. The size of the .DAT file is exactly 128KB less than the size of the .MP4. What's more, the on-SD location of the .DAT is exactly 128KB after the .MP4, which means the two files are actually contiguous with each other. So the camera wrote the .DAT first (while recording the video) but left 128KB before it pre-allocated in the FAT32 cluster table. When the video recording stopped it then wrote the 128KB before the .DAT, which contains all the other MP4 atoms necessary to represent the file, including the 'moov atom' which was reported missing when trying to recover the .DAT as .MP4 earlier using open-source recovery tools (failed because it's missing the full 128KB .MP4 header). The 128KB MP4 container header stuff + mdat atom content represents the full MP4 file. So the .DAT files we're seeing for the interrupted recordings on the R5 are the actual raw video/audio data sans the proper 128KB MP4 header area. On my Canon RP the .DAT directory entry is missing for the interrupt recording. However if I go to the location on the interrupted-SD image as where the .DAT file exists on the good-SD image I see the mdat data. So it appears that on my RP the camera writes to this area during recording without first setting the directory entry to reflect that area is allocated. If you're able to see the .DAT files from your 1DX that means the behavior of that camera is different. Not sure how the R5 acts. Either way, recovering the .DAT file for my RP is trivial because I know what cluster it begins on by comparing it to the good-SD image. I found the .DAT area on the interrupted-SD image and spliced in the 128KB MP4 header from the .MP4 from the good image. The file plays...and I hear a quick snippet of sound, but no video. So there is work to do there. But at least it demonstrates the basic integrity of the full MP4 is in place. To get this working on R5 images we'd have to determine how to build the proper 128KB MP4 container header, the same as how the camera builds it when the user ends a video recording. This might be as simple as splicing from a "good" .MP4 image, but probably requires a lot more effort, and then graft that 128KB header data with the the .DAT file to reconstruct the full .MP4

Thanks for the files. Tried two separate automated MP4 repair programs/scripts, both of which support reading the headers/metdata from the "good" file and attempt to graft that data over those missing elements in the bad file, specifically the missing moov atom in the corrupt file. No joy so far. Oddly when I examine the "good" file the moov atom is actually at the beginning of the file - I expected it to be at the end since that's what's missing from the corrupt file. It might be that Canon relocates the moov atom to the front after a successful recording end. It's usually placed at the beginning of files to facilitate better streaming as the file is being downloaded. Or it might be that IsoBuster didn't recover the file correctly. I'll keep digging. Here's the Atom data from the good file, obtained from AtomicParsley <filename> -T Atom ftyp @ 0 of size: 28, ends @ 28 Atom moov @ 28 of size: 262116, ends @ 262144 Atom uuid=85c0b687-820f-11e0-8111-f4ce462b6a48 @ 36 of size: 65750, ends @ 65786 Atom udta @ 65786 of size: 2056, ends @ 67842 Atom manu @ 65794 of size: 20, ends @ 65814 ~ Atom modl @ 65814 of size: 38, ends @ 65852 ~ Atom urat @ 65852 of size: 16, ends @ 65868 ~ Atom free @ 65868 of size: 1974, ends @ 67842 Atom mvhd @ 67842 of size: 108, ends @ 67950 Atom trak @ 67950 of size: 16328, ends @ 84278 Atom tkhd @ 67958 of size: 92, ends @ 68050 Atom edts @ 68050 of size: 36, ends @ 68086 Atom elst @ 68058 of size: 28, ends @ 68086 Atom mdia @ 68086 of size: 16192, ends @ 84278 Atom mdhd @ 68094 of size: 32, ends @ 68126 Atom hdlr @ 68126 of size: 33, ends @ 68159 Atom minf @ 68159 of size: 16119, ends @ 84278 Atom vmhd @ 68167 of size: 20, ends @ 68187 Atom dinf @ 68187 of size: 36, ends @ 68223 Atom dref @ 68195 of size: 28, ends @ 68223 Atom stbl @ 68223 of size: 16055, ends @ 84278 Atom stsd @ 68231 of size: 311, ends @ 68542 Atom hvc1 @ 68247 of size: 295, ends @ 68542 ~ Atom stts @ 68542 of size: 24, ends @ 68566 Atom stss @ 68566 of size: 296, ends @ 68862 Atom ctts @ 68862 of size: 4472, ends @ 73334 Atom stsc @ 73334 of size: 28, ends @ 73362 Atom stsz @ 73362 of size: 3364, ends @ 76726 Atom co64 @ 76726 of size: 6704, ends @ 83430 Atom sdtp @ 83430 of size: 848, ends @ 84278 Atom trak @ 84278 of size: 8276, ends @ 92554 Atom tkhd @ 84286 of size: 92, ends @ 84378 Atom edts @ 84378 of size: 36, ends @ 84414 Atom elst @ 84386 of size: 28, ends @ 84414 Atom mdia @ 84414 of size: 8140, ends @ 92554 Atom mdhd @ 84422 of size: 32, ends @ 84454 Atom hdlr @ 84454 of size: 33, ends @ 84487 Atom minf @ 84487 of size: 8067, ends @ 92554 Atom smhd @ 84495 of size: 16, ends @ 84511 Atom dinf @ 84511 of size: 36, ends @ 84547 Atom dref @ 84519 of size: 28, ends @ 84547 Atom stbl @ 84547 of size: 8007, ends @ 92554 Atom stsd @ 84555 of size: 103, ends @ 84658 Atom mp4a @ 84571 of size: 87, ends @ 84658 Atom esds @ 84607 of size: 51, ends @ 84658 Atom stts @ 84658 of size: 24, ends @ 84682 Atom stsc @ 84682 of size: 748, ends @ 85430 Atom stsz @ 85430 of size: 6548, ends @ 91978 Atom co64 @ 91978 of size: 576, ends @ 92554 Atom trak @ 92554 of size: 947, ends @ 93501 Atom tkhd @ 92562 of size: 92, ends @ 92654 Atom mdia @ 92654 of size: 847, ends @ 93501 Atom mdhd @ 92662 of size: 32, ends @ 92694 Atom hdlr @ 92694 of size: 33, ends @ 92727 Atom minf @ 92727 of size: 774, ends @ 93501 Atom nmhd @ 92735 of size: 12, ends @ 92747 Atom dinf @ 92747 of size: 36, ends @ 92783 Atom dref @ 92755 of size: 28, ends @ 92783 Atom stbl @ 92783 of size: 718, ends @ 93501 Atom stsd @ 92791 of size: 50, ends @ 92841 Atom tmcd @ 92807 of size: 34, ends @ 92841 Atom stts @ 92841 of size: 24, ends @ 92865 Atom stsc @ 92865 of size: 40, ends @ 92905 Atom stsz @ 92905 of size: 20, ends @ 92925 Atom co64 @ 92925 of size: 576, ends @ 93501 Atom free @ 93501 of size: 168643, ends @ 262144 Atom mdat @ 262144 of size: 730194240 (^), ends @ 0 (^)denotes a 64-bit atom length ~ denotes an unknown atom ------------------------------------------------------ Total size: 730456384 bytes; 66 atoms total. AtomicParsley version: 0.9.0 (utf16) Media data: 730194240 bytes; 262144 bytes all other atoms (0.036% atom overhead). Total free atom space: 170617 bytes; 0.023% waste. Padding available: 1974 bytes. ------------------------------------------------------

Oops, forgot you mentioned raw. I definitely want to focus on processed first. Can you shoot a pair of H.264 4K 24P videos? I'll work on H.264 first then consider H.265 and/or ALL-I

Good. The EXIF can always be grafted with exiftool on but it's probably superfluous for desktop NLE use. Can you possibly share the file via Dropbox or Google Drive? Could you also take another video of the same length that's recorded normally and share that with me as well? I'll be generating my own files too but the process on wiping the cards to perform each experiment is slowing me down so I have free time in between each.