horshack

Thanks. That probably saves me from buying ISObuster for this effort. Was the size of the file about what it should've been for how long the video was? If so then the file may have all the salvageable data recovered and just needs to have the tail of the MP4 container repaired as discussed earlier. I haven't been getting the .DAT but just realized it's because I'm using FAT32 whereas everyone else is likely using exFAT. It defaulted to FAT32 due to the smaller card I was using (32GB), which I chose to make the imaging/recovery experiments faster. If I switch the same card manually to exFAT then the .DAT file starts showing up after an interrupted recording on my Canon RP. I'll be focusing on exFAT in my ongoing work since that's what most are probably using.

It's definitely worth a shot but the evidence suggests that stopping a recording causes the current thermal management state to be written to NVRAM at that point. This is likely why J Marcus only got 5:00 back when pulling the battery - that's because the last NVRAM update was from the previous power-on session where the last video file was saved and the camera reported 5:00 available. So pulling the battery thereafter reverts back to whatever was last saved in NVRAM, which for him was 5:00

It was with an in-camera low-level format, which doesn't actually wipe the entire card because it completes in just a few seconds. After I posted earlier I did another experiment where I completely wiped the card with 'dd' under Linux, then did an in-camera low-level format and repeated the test video w/battery pull. After that IsoBuster only found a single file, named Recovered File 1. mp3 with a size equal to the full capacity of the SD. Which means it's actually less encouraging than before, at least the idea of using an off-the-shelf program rather than engineering a custom solution.

It's definitely a start. The file sizes are off though - it shows only 96K for the MP4 and put the remainder of the card's capacity in bucket labeled .mhk. Not sure if the extraction would successfully reconstruct the file.

Good call @docmoore. Just downloaded IsoBuster and it found the .mp4. It wont let me attempt to extract it without buying a license. I'll search out a few other free/open-source solutions first before biting the bullet on the purchase. Here's a screenshot of the recovery:

I looked into what it might take to recover the lost/missing video file after a battery pull. I reproduced the R5 hack scenario on my Canon RP by recording a minute of video on a freshly-formatted SD card and then pulling the battery while the door-sensor was inhibited. Like what Andrew discovered on his R5 the file is not corrupt but actually completely missing, both when attempting to play the video on the camera and when mounting the card on a computer. I then tried running SanDisk's RescuePro Deluxe and after scanning the entire card it didn't find the video. Here's what I think is happening. Obviously Canon has to write the video data to the card while recording, since it has no other place to retain it beyond the internal SDRAM buffer capacity. You can see easy evidence of this by watching the card access light continuously flicker. The strategy Canon is likely employing is to defer the writing of the official FAT metadata until the recording is orderly stopped. This would include elements such as the FAT directory entry, which anchors the file, and possibly also the linked FAT allocation tables. This would explain why RescuePro couldn't find the file, since there isn't enough or any recognizable metadata to reconstruct the orphaned data sectors associated with the video. Canon is either caching that information in memory until the file is closed or is writing it somewhere on the card that isn't linked to the official FAT structures. The benefit of this strategy, assuming they're using it, is that it prevents any potential filesystem-level corruption from incomplete metadata updates, since none of the official metadata structures are updated to link to the file until it's closed. The downside of this strategy is that it orphans all the data from the file and makes recovery more complicated. There are a few strategies in devising a recovery app for this situation. The first would be to reverse-engineer exactly what if any metadata Canon is writing during the recording and to where. If that orphaned metadata is in the same format as actual FAT structures then it should be relatively straightforward to create a placeholder directory entry and link to it. If the format of that metadata is proprietary to Canon then the process of reverse-engineering its structure would be much more complex. This would all be done by using a sector editor and block-based search tools to compare a freshly-formatted card to one which has a missing video after recording. And lots of effort. I was actually an embedded firmware storage engineer for most of my professional career so this would be in my wheelhouse. I'll see how much demand there might be for this before I spent any serious amount of time on it. In the meantime I'll also try some other recovery apps to see if they have better luck...in case SanDisk's utility is not doing the job it's supposed to. I recommend others try the same so we don't unnecessarily reinvent the wheel 🙂

Good update. FYI, NVRAM refers to any memory that doesn't require the embedded device's primary power source. This can include actual non-volatile memory like EEPROM but is more typically the battery-backed variety, like the Dallas parts that also include an RTC (real-time clock). Here's a sample Dallas datasheet for the latter variety and excerpt: "The DS14285/DS14287 Real Time Clock with NVRAM Control provides the industry standard DS1287 clock function with the additional feature of providing nonvolatile control for an external SRAM. " "The DS14285/DS14287 uses its backup energy source and battery-backup controller to make a standard CMOS static RAM nonvolatile during power-fail conditions. During power fail, the DS14285/DS14287 automatically write-protects the external SRAM and provides a VCC output sourced from its internal battery. "

Someone else did it with a dummy AC battery as well:

Andrew, the video that won't play-back in the camera after removing the battery (without stopping the recording first) is likely recoverable by reconstructing the missing tail data of the MP4 container. I've never had to do this myself but in checking online there are a few open-source solutions, some of which require a dummy "good file" taken anytime from the same camera that they then craft the tail data onto the truncated file. If that solution works then a shooter using the battery hack could just shoot an extra 10-20 seconds of unneeded outro footage that would be tossed from the battery pull.

By overheat do you mean shutdown or do you mean presenting the overheating warning to the user? 4KHQ seems to run over HDMI while the overheat warning is active (per my understanding of mechanicalEYE's post above, so it's not clear what unique significance that warning carries since the camera still operates externally when its active.

I actually wanted to use Horseshit as my online handle around the web but it was either already taken or disallowed by profanity filters 😀 I would speculate it's the combination of the sensor and DIGIC as the main heat sources, in combination with the supporting DDR. The camera can record unlimited 4KLQ, which involves line-skipping/binning instead of oversampling, which reduces the workload for both the sensor (less data sampled per frame) and for DIGIC (less data to process coming off the sensor). The camera can also record long-run times using 4KHQ externally, which means the internal recording path is not used, which means less data flowing through DIGIC for processing for compression. If we presume there's a total operating heat budget and each of these components can contribute to exceed it, it makes sense that there's a matrix of settings (resolution, oversampling vs LQ, compression vs not) which allow the camera to record longer in some combinations and not in others. I'd also speculate the thermal algorithm uses both an instantaneous sampling of current temperature and also a projection of the thermal ramp expected based on the settings used, which in combination result in the projected available recording time that's reported to the user and also used as the thermal shutdown trigger.

Very interesting, thanks for the report. Your observations point to DIGIC's thermals being what the firmware is triggering for the thermal shutdown (ie, the fact the camera allowed itself to get very warm for external recording but then immediately shut down when the camera was configured to allow internal recording, ie CFE card inserted). Also, the fact the camera allows near-unlimited recording over HDMI argues against intentional crippling since external recorders are common for professional video use - if Canon wanted to steer pro video customers away from the R5 to their rumored upcoming RF-based cinema cameras I don't see how they would permit long-run recording over HDMI.

Computers are larger and aren't weather sealed. My laptop stays warm for 10 minutes after I shut it off after it's been running at load.

I have not seen any reports of the camera overheating to the point of shutting down from navigating within the menus or shooting stills. I have seen reports of the cumulative heat from those activities exhausting the video-recording heat budget. The camera can continue to provide video over HDMI during that period so I'm not seeing anything in your comment which refutes how video compression through DIGIC isn't a factor for the overheating that leads to a thermal shutdown.

It's more likely to be DIGIC than CFE. External recording bypasses the entire video compression path, which is very computation and data-intensive (ie, generates a lot of heat). Plus the camera can recorder indefinitely to CFE in 4KLQ.

The crippling has already been disproven (at least the recovery-period aspect of it) - the poster on FM (link) was able to get the full available 8K video recording time after putting the camera in the freezer for just 25 minutes following a thermal shutdown.

1. There will be a point where both DIDIG/SDRAM and internal temps reach apparent homeostasis. Under constant processing load the hottest chips will heat up immediately and reach a plateau quickly, which will be based upon their thermal profile relative to ambient. The ambient temperature around the chips and inside the camera chamber will rise more slowly (hysteresis). As ambient continues to slowly rise the heat-generating chips will get hotter as well but on a slower ramp. This thermal feedback loop will slow over time as the camera's dissipation reaches homeostasis with the heat generation. 2. After powering off the hottest chips will cool immediately, however the ambient temperature around the chips will not. When power and processing load is reapplied those chips will quickly heat back to their pre-shutdown temps since their temp is a function of heat generation relative to ambient temp. 3. What you call "temp inhibitors" is a combination of the latent ambient temperature and logic that is projecting the ongoing heat ramp, which factors in the thermal lag affects described above (hysteresis). The fact the freezer experiment posted on Fred Miranda returned the camera to full video operation times in only 25 minutes serves to both disprove the "timer-based cripple" theory while also demonstrating how the thermal management algorithm is actually working. The logic is serving two purposes - 1) Keep all chips on the PCB within their thermal design limits and 2) Project how much time remains before the chips will reach their thermal limits. The logic for #2 must factor in thermal hysteresis, which to the untrained will may look like intentional crippling and temperature-indifference since the algorithm wont respond as immediately or definitively as one would expect based on instantaneous temps but in reality is actually projecting out a longer-duration thermal ramp. You can see the result of this calculation in the available video-time value reported to the user in the UI.

Thanks. The fact the camera shuts off as soon as you open the battery door means it's using battery-door sensor to perform an orderly shutdown before you get a chance to remove the battery. Unfortunately that makes the experiment more difficult to execute. The only way around it would be to find where the battery door sensor is and manually hold the sensor in while the door is open, then do the video recording, then pull the battery. I think others have tried battery removals without any effect on subsequent available recording times. The difference in this test vs any others I've read is he kept the body in the freezer for 25 minutes. That's much more definitive in disproving the firmware cripple timer theory than putting ice on the camera or adding better heat sinks.

In this test the user was able to get the full 15:00 available following a thermal shutdown after just 25 minutes in the freezer.

Thanks. It's not clear you tried the specific experiment I outlined. The battery should be pulled while the camera is recording internally to the card (abrupt shutdown), somewhere near where it would thermally shutdown. Then the battery should be put back in and check what the available recording time the camera reports.

On previous Canon bodies the temperature report in the EXIF is from what's called the "EFIC temp" (source), which is a secondary, low-speed chip responsible for the lens interface and speedlites. Canon actual documented this chip in their 1DM3 technical whitepaper (page 33), where Canon refers to it as follows "EFIC as the interface for the lens and external Speedlite" and explicitly states it's located on the camera control board, separate from the image sensor and DIGIC. If the R5 follows this arrangement this means the temperature in the EXIF is not of the sensor or DIGIC, which explains why it plateaus quickly and doesn't correlate to the video-recording thermal shutdown events, which instead are likely related to the temperature of DIGIC and/or the surrounding support chips like DDR.

Try a battery pull while the camera is still on, both just after stopping a record and if that doesn't extend the subsequent recording time, while the video is actually recording. Check if the camera "remembers" the remaining time limit available when you plug the battery back in (details here).

The best way to determine if the time limit is firmware-related is to attempt to defeat the firmware's mechanism to record the time/states associated with the thermal shutdown. All camera settings and firmware state variables are stored in NVRAM. The settings are typically only written out when the camera does an orderly shutdown (optimization). You can verify this yourself by changing one of the shooting parameters (aperture, shutter speed, ISO) and then pulling out the battery while the camera is on. When you plug the battery back in you'll likely see the camera didn't save that change. The state variables associated with any potential video timer/limit/thermal threshold might be defeated using this same technique.

"Why does the Canon API documentation claim you are able to poll the temperature, but the camera only returns a text status like “Normal” rather than a real temperature in Celsius? Why is this all-important temperature readout obscured?" The precise internal temperature is available in the EXIF metadata of R5 images. If you'd like to experiment on thermal shutdown temperature thresholds/cooldown periods you can take a photo before/during/after the test and use the EXIF of those photos to monitor the temperatures over the period of the test. For example, here's an exiftool invocation to get the internal camera temperature for all R5 .CR3 raw files in a directory: exiftool.exe -CameraTemperature "c:\mypics\*.cr3"

Agree, the camera market has been collapsing for some time and I think Canon just chose to limit its R&D investment to maximize its profit and keep the cash-cow going for as long as they could. They might have been able to squeeze some additional unit sales by targeting hybrid/video shooters with less-crippled offerings but they probably decided it wasn't useful since the market is seen falling into a hyper-competitive, lower-margin business anyway, with insufficient volume on the low end to amortize their R&D and manufacturing capital investments. Such is not the case for their professional video division so crippling the video on the prosumer/enthusiast offerings was probably seen as an additional safeguard against cannibalization of sales of that division.