Petition for Samsung NX1 hack

[Syme]

1 hour ago, tugela said:

The sensor can do full reads at 240 fps, so it isn't an issue with the sensor. The reason for higher fps not being enabled is almost certainly because the processor can't do it or becomes unreliable due to bandwidth issues. The sensor itself is capable of doing 6K and any fps up to 240.

What comes out of live view is not RAW video. It is uncompressed, but still processed. It will still be 8 bit 4:2:0.

http://www.samsung.com/semiconductor/products/cmos-image-sensor/camera-cis/S5KVB2?ia=218

Says 25fps for the full sensor. That means about 9ms per line, which results in about 30ms minimum frame time for a 16:9 crop. Hence the 30fps maximum for UHD and 30ms rolling shutter that was measured by both Cinema5D and some guy on DVXuser. That also lines up with the 1080p rolling shutter and maximum frame rate measured in various places. The predicted speed in the closest binning level to 720p is also about 200fps based on that time per line, which is what Cinema5D saw on the pre-production NX500.

Continuous shooting at 15fps is also consistent with that readout speed.

Yes, I know a Samsung marketing guy said otherwise in an interview, but I'm not convinced he really knew what he was talking about. I think it is more likely that after a long corporate game of "telephone" the actual numbers got mistaken.

It's the word of one marketing manager in an interview vs the official product information published by Samsung Semiconductor and all the specifications and measurements taken of the camera that match that number.

The Nikon j5 is a good example of what it actually looks like when a fast sensor is limited by a slow processor. The sensor does over 5k at 60+fps, but the processor will only handle UHD at 15fps. However the rolling shutter is still excellent, since the first step of the image processing pipeline is to just dump the raw readout to RAM, which is more than fast enough. It can do continuous shooting at the full speed of the sensor since it doesn't use a mechanical shutter in that mode, and they didn't decide to limit it.

I see no reason for Samsung to go to all the effort to build a CMOS sensor capable of 10x the clock speed they run it at in their flagship product, then lie about the actual capabilities on the product page. Furthermore there is no CMOS circuit I have ever heard of that can be overclocked by 1000%, even under laboratory conditions (liquid nitrogen, etc), so I am somewhat skeptical that they ran the thing at 240fps in testing.

The live view is derived from the raw sensor data. The question is where the boundary between the ARM cores and the special purpose hardware is in the image processing pipeline. Magic Lantern RAW video works because they found that Canon's live view mode puts the raw sensor data somewhere the general-purpose processor can access it before de-bayering it and sending it to the screen or to HDMI as RGB video. Nikon does it the same way, as shown by the Nikonhacker developers. I don't know how Samsung does it, but it is quite likely to be like Canon and Nikon.

[Syme]

I just realized the Tizen git repository (https://review.tizen.org/git/) includes open-source code for libmm-camcorder, which seems to have all the same functions as libmmf-camcorder.so on the NX1. Not sure what the difference is. Looks like this is a goldmine for understanding what the camera app is doing. I'm glad I found it before I started digging into the libmmf-camcorder.so binary. There is also a repository for a default camera app, which doesn't seem to be the same as on the NX1, but if we're lucky they might be closely related.

[tugela]

6 hours ago, Syme said:

http://www.samsung.com/semiconductor/products/cmos-image-sensor/camera-cis/S5KVB2?ia=218

Says 25fps for the full sensor. That means about 9ms per line, which results in about 30ms minimum frame time for a 16:9 crop. Hence the 30fps maximum for UHD and 30ms rolling shutter that was measured by both Cinema5D and some guy on DVXuser. That also lines up with the 1080p rolling shutter and maximum frame rate measured in various places. The predicted speed in the closest binning level to 720p is also about 200fps based on that time per line, which is what Cinema5D saw on the pre-production NX500.

Continuous shooting at 15fps is also consistent with that readout speed.

Yes, I know a Samsung marketing guy said otherwise in an interview, but I'm not convinced he really knew what he was talking about. I think it is more likely that after a long corporate game of "telephone" the actual numbers got mistaken.

It's the word of one marketing manager in an interview vs the official product information published by Samsung Semiconductor and all the specifications and measurements taken of the camera that match that number.

The Nikon j5 is a good example of what it actually looks like when a fast sensor is limited by a slow processor. The sensor does over 5k at 60+fps, but the processor will only handle UHD at 15fps. However the rolling shutter is still excellent, since the first step of the image processing pipeline is to just dump the raw readout to RAM, which is more than fast enough. It can do continuous shooting at the full speed of the sensor since it doesn't use a mechanical shutter in that mode, and they didn't decide to limit it.

I see no reason for Samsung to go to all the effort to build a CMOS sensor capable of 10x the clock speed they run it at in their flagship product, then lie about the actual capabilities on the product page. Furthermore there is no CMOS circuit I have ever heard of that can be overclocked by 1000%, even under laboratory conditions (liquid nitrogen, etc), so I am somewhat skeptical that they ran the thing at 240fps in testing.

The live view is derived from the raw sensor data. The question is where the boundary between the ARM cores and the special purpose hardware is in the image processing pipeline. Magic Lantern RAW video works because they found that Canon's live view mode puts the raw sensor data somewhere the general-purpose processor can access it before de-bayering it and sending it to the screen or to HDMI as RGB video. Nikon does it the same way, as shown by the Nikonhacker developers. I don't know how Samsung does it, but it is quite likely to be like Canon and Nikon.

That is the nominal specification for a sensor available for general sale (not necessarily the sensor in the NX1/500). Presumably those specs are based in conjunction with the capabilities of the associated processors, which in turn determines the nominal specification.

Components usually can be run well in excess of their nominal specs, especially considering that the power requirements for the sensor are half of what conventional sensors running at half the read speed require. Plus, that applies to 12 bit data, not 8 bit. So you have quite a bit of room to run the sensor well beyond the nominal specs, which is apparently what they did in debugging.

The NX1 does a full sensor read without line skipping/binning at 60 fps in FHD (we know this, because otherwise light sensitivity would decrease in FHD relative to UHD, and it does not), so obviously the sensor can be read fully at least at those speeds. Not to mention 120 fps, which is also supported in the camera. It is not reading only parts of the sensor, it is reading all of the sensor and at least up to 120 fps. What the camera can't do is process all that data in UHD mode, hence the frame rate limitation and rolling shutter issues.

The DRIMe V/s processors are not mentioned on that site for example. What other people can buy is not necessarily what they use in their own products.

As I said before, live view is not RAW.

[tugela]

 

One more thing, regarding rolling shutter. The NX1 has none when shooting 6K stills at 15 fps either. So obviously the argument you are making regarding the read speed of the NX1 sensor cannot be correct.

The difference between video and stills burst is how the data is processed. For video the NX1 is likely treating the data as a serial stream and processing it as it arrives. That would go in two phases, firstly debeyering and scaling to generate the pixels and then the processing to generate the compressed frame. If the processing step is rate limiting then for a 30p frame the time required would be 1/30 seconds, or 33ms (this probably also sets limits on the bit rates btw). Rolling shutter would have the same value as a result, since the pixels are being streamed in serially. This is what is happening with UHD. For FHD however, the raw data is assembled into 1/4 of the number of pixels. Since there is 1/4 of the data to be processed, the frame can be assembled in 1/4 of the time, in other words 8.3 ms. The rolling shutter measured for 1080p is 7.9 ms. That means that rolling shutter is determined purely by the processing time in the NX1, which in turn means that the maximum frame rate the sensor is capable of is well in excess of any of the frame rates available in video in the camera.

Since the image can be processed in 8.3 ms in 1080p, it means that only 1/4 of the available processing time is being used at 30 fps when in FHD mode. If you use all of the time available you should be able to do 120 fps, and we see exactly that mode present in the NX1.

On the other hand if you look at stills, burst mode assembles all of the raw data in the buffer first, then (at a later time) processes the data into the image. In that sense it behaves like a global shutter and you see little or no rolling shutter artifacts.

What all of that means is that the NX1 has to be reading the sensor at speeds greater than 120 fps.

The marketing dude you referred to is Jay Kelbey, who has given a number of interviews about the NX1. He seems to know what he is talking about and clearly states in all of them that the camera does full sensor reads at 240 fps. Apparently it is used for object tracking in the camera, and has a dedicated hardware block in the processor for that purpose. He also says that the engineers prepared 28 mpixel videos at 240 fps when they were debugging that function.

Since both UHD and FHD are processor limited, it implies that the camera cannot do more than 30 fps in UHD, or 120 fps in FHD. In order to shoot at higher frame rates the processor has to be improved in some way, either by a later generation faster version, or by overclocking it.

[Geoff CB]

1 hour ago, tugela said:

 

One more thing, regarding rolling shutter. The NX1 has none when shooting 6K stills at 15 fps either. So obviously the argument you are making regarding the read speed of the NX1 sensor cannot be correct.

This is false, the NX1 like most modern mirrorless have a mechanical shutter that prevents the rolling shutter artifact. You can see the rolling shutter when you shoot in silent mode on cameras like the A7s which reads the data off the sensor, but in regular mode it is not there.

[Marco Tecno]

Not sw related but interesting for the nx1 community:

 

http://www.dpreviewer.net/forums/redirect-unread/3818034?utm_campaign=internal-link&utm_source=footer-logo&utm_medium=image&ref=footer-logo

 

Nobody able to start from here and design a canon ef to nx adapter?

 

https://www.google.it/url?sa=t&source=web&rct=j&url=http://web.media.mit.edu/~bandy/invariant/move_lens.pdf&ved=0ahUKEwjqt8uVgYTLAhVI0w4KHQq-Dn8QFggeMAE&usg=AFQjCNGD_oK2ET2CqvObaOtNCS2dS39qzQ&sig2=WNXWqlcFb0STLCWA8xaamg

 

I'm 100% sure that:

1) this can be done

2) could be done with an arduino board in the middle, but a bit impractical (anyway I'd still want this) and a transformer for voltage

3) could be done with a custom designed chip, but is has to be designed and...manufactured

[vaga]

40 minutes ago, Marco Tecno said:

Not sw related but interesting for the nx1 community:

 

http://www.dpreviewer.net/forums/redirect-unread/3818034?utm_campaign=internal-link&utm_source=footer-logo&utm_medium=image&ref=footer-logo

 

Nobody able to start from here and design a canon ef to nx adapter?

 

https://www.google.it/url?sa=t&source=web&rct=j&url=http://web.media.mit.edu/~bandy/invariant/move_lens.pdf&ved=0ahUKEwjqt8uVgYTLAhVI0w4KHQq-Dn8QFggeMAE&usg=AFQjCNGD_oK2ET2CqvObaOtNCS2dS39qzQ&sig2=WNXWqlcFb0STLCWA8xaamg

 

I'm 100% sure that:

1) this can be done

2) could be done with an arduino board in the middle, but a bit impractical (anyway I'd still want this) and a transformer for voltage

3) could be done with a custom designed chip, but is has to be designed and...manufactured

I think it is probably relatively easy to design an adapter and 3D print it so that the flange distance and where the lens sits is the same. I could probably try it in the summer. Of course, making a smart adapter is a little more complicated but I too have been wondering why that hasn't happened for the nx1 at all.

[Syme]

16 hours ago, Syme said:

Says 25fps for the full sensor. That means about 9ms per line, which results in about 30ms minimum frame time for a 16:9 crop

That was supposed to be 9us, not 9ms. 9ms would mean 0.03fps...

Also I'm not even going to bother arguing with people who still think the NX1 can do a 240fps readout of every individual pixel. That's just not what I'm here to do.

[mercer]

I wonder if Samsung has a firmware version of the NX500 preproduction 2.5k lying around? If enough people request that they offer that version to their current and future Samsung brand customers, they may just release it. And who knows what else they had in the firmware pipeline, for the NX series, before they decided to call it quits. I haven't been paying attention to this thread too much, so excuse me if this has been discussed, but does anyone here have any Samsung contacts? 

3 hours ago, vaga said:

I think it is probably relatively easy to design an adapter and 3D print it so that the flange distance and where the lens sits is the same. I could probably try it in the summer. Of course, making a smart adapter is a little more complicated but I too have been wondering why that hasn't happened for the nx1 at all.

It hasn't happened for the same reason Samsung is getting out of the camera business... The demand is not there. 

[tugela]

7 hours ago, Geoff CB said:

This is false, the NX1 like most modern mirrorless have a mechanical shutter that prevents the rolling shutter artifact. You can see the rolling shutter when you shoot in silent mode on cameras like the A7s which reads the data off the sensor, but in regular mode it is not there.

In silent mode the A7 is shooting its frame as if it was a video frame, rather than the way it would if it were a still frame, hence the rolling shutter differences. If your shutter speed is 1/60 sec having a mechanical shutter or not should have no impact on the presence or absence of rolling shutter. When you shoot in stills mode the raw data goes into the buffer and then gets processed later, but when shooting in video mode the raw data goes directly into the image processor hardware. It is fundamentally different.

This is one of the reasons why cameras like the 5D series (and other Canon DSLRs) can be hacked the way they are. The video in those cameras is assembled in the buffer as if they were stills, so you have discrete frames to with. In Canon's video cameras you probably can't do the same thing since the data goes directly into the dedicated signal processing hardware of the Digic DV processor, and there is no discrete raw frame to intercept.

 

[Syme]

tl;dr: 2.5k mode may be lurking just under the surface. Not sure, but it looks promising.

Did some more digging today and I'm pretty sure the valid resolutions one the NX1 and NX500 are defined at least in part by the file /usr/etc/mmfw_camcorder_dev_video_pri.ini. It can be found in both the rootfs filesystem and the opt filesystem images (under etc in opt).

I found that a while back but I was sure it couldn't be the right file since the defined name and comments for the file say it is for a Fujitsu cellphone camera. Furthermore the stills resolutions listed are completely wrong for the NX1/500. However I spent a while reading the source code for libmm-camcorder.so, and it reads the config values from those files. I'm still not sure how it determines the correct sensor and image processor parameters for a given mode, however, since there are several combinations that would produce each resolution, and the details aren't in the configuration files I have seen.

However the interesting part is that the one difference between mmfw_camcorder_dev_video_pri.ini on the NX1 and that same file on the NX500 is that the NX500 includes an additional "2560,1440" mode. That's good news since it seems to indicate that they didn't bother to remove the 2.5k mode from the underlying libraries and instead just got rid of it in the camera app. For what it's worth the "QHD" icon is still present in the resources folder for the camera app, so it looks like they weren't very thorough in getting rid of it. 

[mercer]

29 minutes ago, Syme said:

tl;dr: 2.5k mode may be lurking just under the surface. Not sure, but it looks promising.

Did some more digging today and I'm pretty sure the valid resolutions one the NX1 and NX500 are defined at least in part by the file /usr/etc/mmfw_camcorder_dev_video_pri.ini. It can be found in both the rootfs filesystem and the opt filesystem images (under etc in opt).

I found that a while back but I was sure it couldn't be the right file since the defined name and comments for the file say it is for a Fujitsu cellphone camera. Furthermore the stills resolutions listed are completely wrong for the NX1/500. However I spent a while reading the source code for libmm-camcorder.so, and it reads the config values from those files. I'm still not sure how it determines the correct sensor and image processor parameters for a given mode, however, since there are several combinations that would produce each resolution, and the details aren't in the configuration files I have seen.

However the interesting part is that the one difference between mmfw_camcorder_dev_video_pri.ini on the NX1 and that same file on the NX500 is that the NX500 includes an additional "2560,1440" mode. That's good news since it seems to indicate that they didn't bother to remove the 2.5k mode from the underlying libraries and instead just got rid of it in the camera app. For what it's worth the "QHD" icon is still present in the resources folder for the camera app, so it looks like they weren't very thorough in getting rid of it. 

Good job. If you guys can unlock the 1440p... I'll just go ahead and use it. 

[Geoff CB]

38 minutes ago, Syme said:

tl;dr: 2.5k mode may be lurking just under the surface. Not sure, but it looks promising.

Did some more digging today and I'm pretty sure the valid resolutions one the NX1 and NX500 are defined at least in part by the file /usr/etc/mmfw_camcorder_dev_video_pri.ini. It can be found in both the rootfs filesystem and the opt filesystem images (under etc in opt).

I found that a while back but I was sure it couldn't be the right file since the defined name and comments for the file say it is for a Fujitsu cellphone camera. Furthermore the stills resolutions listed are completely wrong for the NX1/500. However I spent a while reading the source code for libmm-camcorder.so, and it reads the config values from those files. I'm still not sure how it determines the correct sensor and image processor parameters for a given mode, however, since there are several combinations that would produce each resolution, and the details aren't in the configuration files I have seen.

However the interesting part is that the one difference between mmfw_camcorder_dev_video_pri.ini on the NX1 and that same file on the NX500 is that the NX500 includes an additional "2560,1440" mode. That's good news since it seems to indicate that they didn't bother to remove the 2.5k mode from the underlying libraries and instead just got rid of it in the camera app. For what it's worth the "QHD" icon is still present in the resources folder for the camera app, so it looks like they weren't very thorough in getting rid of it. 

God I wish they had kept the 2.5K mode. Hope it can be enabled, would it be possible to buy a NX500 and test this?

 

[SMGJohn]

On 18.2.2016 at 6:10 PM, MountneerMan said:

 

I was under the same impression.

@SMGJohn et al, where can I download the SKD and 91 page PDF? I would be interested in giving it a look over. USB 3.0 has a transfer rate of 5Gb/sec witch is more than enough for a 4K RAW stream. I am imagining being able to plug in a laptop(with an SSD) via USB and using it like an external recorder except you would have pretty much complete control of the camera(except zoom and aperture depending on the lens) as well.

Does any one think this would even be possible? I am not a CS expect.

 

Did you read the dpreview posts I linked yesterday above? otto K said he thinks its possible to run scripts from the SD Card witch would allow for the use of debuggers.

Also because the NX1/NX500 are Tizen based it sounds like it could be quite easy(relatively speaking) to create an emulator https://developer.tizen.org/development/tools/common-tools/emulator . Again I am far from a CS expect and I am way out of my depth and really have no idea. Anyone who knows better please correct me.

You can request the SDK here http://nxcameraportal.cloudapp.net/SDK/UserInfo or through the i-Launcher application, I requested a copy now I just have to wait, apparently the SDK gives you control over the camera completely for development of software.

I have tried finding external download links for it, but no imbecile has bothered sharing it apparently. Talk about being greedy, I seen loads of people with the SDK tool but no one gives download link and their posts are usually months old.

[giannis_ch]

Is the SDK you are looking for here https://opensource.samsung.com/reception/receptionSub.do?method=sub&sub=T&menu_item=photography&classification1=digital cameras ?

[kidzrevil]

damn sounds like we can make a breakthrough any minute now !

[Syme]

19 minutes ago, kidzrevil said:

damn sounds like we can make a breakthrough any minute now !

Don't expect anything from me in the near future. I'm pretty swamped with homework and tests to study for at the moment 

As much as I enjoy messing around with the NX1 firmware, I've got to remember my priorities...

[ESGI Media]

Speaking of features, the NX1 actually records metadata in the video files, as described in this video. Would be nice if there was an option to have it written to the video files in a different format so it is easily digested by NLE's.

[undecided]

21 hours ago, SMGJohn said:

You can request the SDK here http://nxcameraportal.cloudapp.net/SDK/UserInfo or through the i-Launcher application, I requested a copy now I just have to wait, apparently the SDK gives you control over the camera completely for development of software.

I have tried finding external download links for it, but no imbecile has bothered sharing it apparently. Talk about being greedy, I seen loads of people with the SDK tool but no one gives download link and their posts are usually months old.

Thanks, I just requested it as well.

15 hours ago, giannis_ch said:

Is the SDK you are looking for here https://opensource.samsung.com/reception/receptionSub.do?method=sub&sub=T&menu_item=photography&classification1=digital cameras ?

The one you're pointing to is pretty locked down. The important stuff comes in binary format not source code.

[Chant]

New to the forum, but I have been working on the opening of the firmware for quite a few months. The second I heard about this camera coming from a modded gh2 I knew things were going to be fun.

I had plans to release what I found after having a stable custom build.

Good news after reading this thread as I seem to have quite a bit more discovered than most.

-bootloader

-3d lut info

-rgb888 info

And quite a few more things that make this an ideal camera to mod.

In the interview they basically implied they either had asics or fpgas built along side the arm cores.

"DE: When we were talking previously, you said that the architecture of DRIMe V is very different than DRIMe IV. What sorts of changes are made between them?

JK: The biggest change is the structure of the Image Signal Processor. The DRIMe V ISP is very different. Most ISPs have key parts of the processing hardwired to get the needed speed. What's really new with the DRIMe V is that the "hardwiring" can be reconfigured.

DE: That sounds very significant, although I have to admit I don't know how common it is that ISPs are actually hardwired.

JK: There's not much variability in them. Sometimes you'll get thresholding changes and small numeric variables, but the image path is usually very locked down...

DE: It's pretty much, a fixed pipeline.

JK: With a few tuning pieces, yeah.

DE: Whereas this is more configurable.

JK: Yes.

DE: And it's configurable not at the level of changing a firmware program, but you're actually changing the hardwiring. There are switches you can use to change the configuration of the functional units?

JK: Yeah, basically. And Samsung's software people are all over it; they're pretty sharp and I believe that we're really going to be able to tap fully into the DRIMe V's capabilities for the NX1.

DE: So even though it's hardwired, some of the connectivity is programmable through the firmware, so we can actually look for continued "firmware" development that would continue to improve performance, including the parts that are implemented in hardware.

JK: Yes.

DE: So there may be significant improvements in the near future. That's pretty interesting, because it sounds like it's already starting out pretty capable.

JK: And not just image quality, there's a lot going on in terms of programmability, like we talked about Samsung Auto-Shot mode before. Here's the UI for it."

TLDR: fpga or cpld or asic that can be reconfigd.

Now a few years ago samsung released info about adding fpgas into asics http://www.eetimes.com/document.asp?doc_id=1200317
 

And have teamed up with Lattice on a few projects http://www.oregonlive.com/silicon-forest/index.ssf/2014/04/which_oregon_company_has_a_chi.html

The back end camera control files seem to be written in .cpp so diving deep into the firmware to do a comparison of hex/string/hash changes and tracking the changes across the updates is what I am doing right now. And seeing which values can be modded. Recompiling is a different story but what it seems to be is that building a smiliar man in the middle to what magic lantern would be possible. Have a gui that would simialr to ptools in camera using the web browser capabilities.

Now this is not my full time work so Ive been progressing slowly as I have many projects on the go.
This is also coming from someone who doesnt yet have an nx1. Doing the proof of concept and running the fw in a vm to emulate the possible bricking before things are bought.

I am also tracking similarities to the nx500 to see what things are possible in that camera. Nice b camera which should be able to compete with a shortening of its life but at the cost 10x performance is worth the lifespan limit imo.