HOW TO: Upscaling the first 1896 Lumière Brothers film recordings to full colour 4K/60p with neural network engine

[Andrew Reid]

Shot by the Lumière Brothers in 1896 this is some of the first footage committed to film. There is now a technique to upsample this footage to high resolution 4K/60p using neural networks. Here it is in action on the piece “Arrival of a Train at La Ciotat”.

Read the full article

[theraywong]

Just pointing out a typo in the first sentence, 1986 instead of 1886. Not hating, just thought you may want to know. Weird how they've made it 60fps, giving the vintage footage that soap opera/videol look

[sanveer]

1 hour ago, Andrew Reid said:

Shot by the Lumière Brothers in 1986 this is some of the first footage committed to film. There is now a technique to upsample this footage to high resolution 4K/60p using neural networks. Here it is in action on the piece “Arrival of a Train at La Ciotat”.

Read the full article

Saw this yesterday somewhere, and it's absolutely amazing. The grading is a little curious, but the digital restoration is superb. I guess it could be upressed to 8k too, if necessary. 

[smithstone7]

This is the actual source video used as the input, it is much superior to the output (so far) of these neural networks: 

 

[androidlad]

 

[chmop]

The best way to have a clean 4k would be to... use the 4k master the "Institut Lumière" did in 2015 for the 120th anniversary

The "original" link in the colored version is misleading. It is NOT the original used to upscale to 4k. The right link is in the B&W 4K/60p video before colors.
They upscaled the 1080p from the bluray "Lumière ! L'aventure commence" (that itself has been done with the new 4k master!)
Uncropped YT version : https://www.youtube.com/watch?v=ymjlHsPmesk
Trailer with english subtitles here : https://www.youtube.com/watch?v=BVcRXIN_woc (it seems it's on Amazon Prime in some countries (I don't have it here in France))

I really don't like the 4k/60p upscale... I think it's worse than the clean 1080p original. I don't see any interesting details. I feel like it mostly adds ugly artefacts everywhere.
The AI part is often just a semi-bullshit marketing term. Topaz Gigapixel AI is just another commercial "AI" upscaler. Everybody has one nowadays. Even my new cheap tiny TV box thingy (Shield TV) has an integrated 4K upscale with an "AI enhanced" label on it. I'm not sure where the AI comes in that.  I'm pretty sure it doesn't use any neural network/tensors/anything in realtime and doesn't learn when I feed it videos 
For the 60p: it doesn't help where it could (where the original "jumps" sometimes).
And it's badly cropped.

Colorization with DeOldify is still impressive but still far from perfect. It still has those colors leaks, and was clearly not made for videos.

That's not much than a fun experiment, for me 😕

[no_connection]

This is the source they say they used for the original, not the one you linked.

They do link the other one as "original" to look more impressive.

The coloring is cool but it's not really much better that the original imo. Sharper edges tho.

*edit* It does add some detail but it's spotty on what it decides to enhance. This is the first frame.

[noone]

So what program should I get/use to upscale video from my late Dads ancient Canon G10 that only does SD at best (but has really nice stills at base ISO and inbuilt ND filter).

[Waynes]

Lol.  It almost looks rolling shutter on that train.  How they decide the color?  That is one of the difficult things in the industry.  I was looking at doing a technique to extract original colors in the 80's or 90's, but the information needed wouldn't survive the post process.  But the his AI thing is a bit garbage.  There are deterministic ways of restoring film for decades, and I looked at doing so in the 1990's/maybe 1980's.  If only they had a prism splitter to put little primary colored images of film in between the sprocket holes that would help colorisation and could have been used as the basis of future primitive colour projection grading.  But, then again, I'm over one hundred years ahead of them.  You know, they could have simply done a long frame split up three or more times to use a splitter to give the primary colour very accurately in the 1800's.  I wonder if that would actually enhance film now?  Extra wide formats are used regularly now, that's nearly a 4:3 sideways frame.  IMax of the old days, OMax.

And I don't even work at Red! :)

[Wild Ranger]

The upscaling software is Topaz gigapixel for video, this software is in current closed beta for developers and testers to use, is not in the web page.

But, since I'm in the beta test page, i can provide a link:

https://topazlabs.s3.amazonaws.com/packages/win/videoenhanceai/VideoEnhanceAI-1.0.1-windows-x64-installer.exe?fbclid=IwAR2fORNpdKqvFvleUeH_xkvNOfmz3c92tjigWOwcMRsKrb5-WP6NseMlaWE

Bye!

[Waynes]

How come they.dudnt do it in 16k 120fps, or 32k?  People in the know, will know what I'm hinting at.  It's easy to upscale by making a fine delimitation point around a pool of visual information (boundaries, including hard edges but also soft edges in toning movement).  You apply the same approximation of boundaries to faster frame rates to produce frames inbetween and in the existing ones   This allows you to go to  whatever K resolution and frame rate, but lacks the missing detail.  However, it's hard to use regular AI well.  They waste huge amounts of energy for more advanced routines, which could be replaced by economic procedural programming.

 

Now, Intel had a nice upscaling routine many years back.  It used to be a hour or so, or a hundred hours a frame (I forget which?).  With today's hardware and even custom hardware, that could be much faster.  Now, onto modern TV AI upscaling.  I have read comments it is very cartoonish.  Another accidental upscaler is fractal video codec.  Anyway, the upscaler I envisionaged doing, would be designed to restore real detail quickly.  So, thanks for the memories here (maybe me grandfather in on the train platform) but I'll pass on this as far as real detail goes.

 

An interesting thing noted on the video comments, is that the train color is wrong, and it should be a more vibrant green.  Setting aside the possibility this may actually be a color deviation from the normal color, in real life, some monochrome film had problems with various color.  This film might have had problems strongly registering the bright green.

 

[Waynes]

Here's an idea.  Using a beam primary splitter to three vertically stacked frames idea above, could a superior picture to modern film.  This, even without customising the chemical formulation of each primary sub frame in the frame.  Film is heavily formularised in layers with HDR tricks.  I imagine film production buffs in big studios might like to spend the extra film in order to produce a feature that matches any current cinema popular camera plus some more?

[Andrew Reid]

Good news for us, we can now sell everything and shoot on an iPhone knowing that somebody in 100 years will neural-network the footage to make it look like its shot on an Alexa.

[Mokara]

21 hours ago, Waynes said:

Lol.  It almost looks rolling shutter on that train.  How they decide the color?  That is one of the difficult things in the industry.  I was looking at doing a technique to extract original colors in the 80's or 90's, but the information needed wouldn't survive the post process.  But the his AI thing is a bit garbage.  There are deterministic ways of restoring film for decades, and I looked at doing so in the 1990's/maybe 1980's.  If only they had a prism splitter to put little primary colored images of film in between the sprocket holes that would help colorisation and could have been used as the basis of future primitive colour projection grading.  But, then again, I'm over one hundred years ahead of them.  You know, they could have simply done a long frame split up three or more times to use a splitter to give the primary colour very accurately in the 1800's.  I wonder if that would actually enhance film now?  Extra wide formats are used regularly now, that's nearly a 4:3 sideways frame.  IMax of the old days, OMax.

And I don't even work at Red!

It is not rolling shutter, it is the film flexing as they are cranking it past the shutter.

[Mokara]

On 2/9/2020 at 4:07 AM, theraywong said:

Just pointing out a typo in the first sentence, 1986 instead of 1886. Not hating, just thought you may want to know. Weird how they've made it 60fps, giving the vintage footage that soap opera/videol look

The real original would not have had a fixed frame rate. Certainly nothing that would be compatible with anything today, so all modern frame rates would be "soap opera/video" with this footage, lol.

[JR Lipartito]

On 2/10/2020 at 12:48 PM, Wild Ranger said:

The upscaling software is Topaz gigapixel for video, this software is in current closed beta for developers and testers to use, is not in the web page.

But, since I'm in the beta test page, i can provide a link:

https://topazlabs.s3.amazonaws.com/packages/win/videoenhanceai/VideoEnhanceAI-1.0.1-windows-x64-installer.exe?fbclid=IwAR2fORNpdKqvFvleUeH_xkvNOfmz3c92tjigWOwcMRsKrb5-WP6NseMlaWE

Bye!

@Wild Ranger I don't suppose you've got a handy link like that for Mac, do you?

[ninetto]

Jeeez-Louizzz, can't the computer nerds leave anything alone? Of course they shortened the other film, WORKERS LEAVING THE FACTORY because it seems the attention-span on internet, even with the attrocious silly-yee-oldie soundtrack added, simply is not adequate enough to reflect on what one is seeing... or not.

Illusion fetishism at its worst.

 

 

[Wild Ranger]

13 hours ago, JR Lipartito said:

@Wild Ranger I don't suppose you've got a handy link like that for Mac, do you?

Sorry, nope.

But today the released the public version, there's a trial.

https://topazlabs.com/video-enhance-ai/?utm_source=Gigapixel+AI+for+Video&utm_campaign=fbf309c150-EMAIL_CAMPAIGN_2020_01_02_05_52_COPY_01&utm_medium=email&utm_term=0_eb14b23af4-fbf309c150-92946301&mc_cid=fbf309c150&mc_eid=2d27f88d69

[JR Lipartito]

13 minutes ago, Wild Ranger said:

Sorry, nope.

But today the released the public version, there's a trial.

https://topazlabs.com/video-enhance-ai/?utm_source=Gigapixel+AI+for+Video&utm_campaign=fbf309c150-EMAIL_CAMPAIGN_2020_01_02_05_52_COPY_01&utm_medium=email&utm_term=0_eb14b23af4-fbf309c150-92946301&mc_cid=fbf309c150&mc_eid=2d27f88d69

Thanks for the heads up! Although looks like it only works with Nvidia graphics cards so still not on for Macs...

[PaulUsher]

Magical footage. I always loved hearing that audiences freaked out as the train rolled ‘at’ them (could be one of those apocryphal stories)

fun fact: the Lumiere bros famously bitched about the crop and the glitchy autofocus on their speedbooster