High frame rate video - audio possibilities?

[matthere]

Hi folks, I am looking into the potential of processing audio at the time of recording, it's for an art project where I'd like to slow down an audio signal in a live situation.. 

I am looking to find out if there might be a way to cut up a live audio signal and feed part of the signal in a way that it could be monitored in a meaningful way?

I understand I might need to change the pitch and possibly gate the sound some way that allows only a quarter of the sound's duration to be monitored?

Any suggestions gratefully received, thanks ?

[IronFilm]

Think about it: 

Is it possible to watch slow motion fully live while it is being recorded?

Same principle applies to audio.

[kye]

Here's a thought - every second you take the first 1/5th of a second and slow it down to 20% speed and watch that for the full second, then at the start of the next second you skip forward the other 80% of the second to get to the start of the next second and the cycle starts again.

It would be semi-realtime, being that people would see and hear themselves in slow-motion with only a second delay.  

Considering you haven't specified what you're trying to accomplish, I'm not sure how this suits, but if you're looking at an art installation or something then that would work  

[matthere]

4 hours ago, IronFilm said:

Think about it: 

This has been on my mind for some time.. You probably understand artists don't think straight.. ?

As Digital Signal Processing increases we are seeing quite a lot of realtime audio processing, guitar effect type stuff and things like granular synthesis of live sound.. I was wondering if anyone might have come across any methods for time effects on audio used in this way.

As mentioned by @kye sound might be able to be chopped up.. is there a way to only monitor part of the live sound, I am wondering if there is a way of getting rid of sections of sound, and just listening to short sound parts as you go along?

[matthere]

A crap sketch of the general idea..

But still trying to understand this in my own head, so I maybe completely off the mark as to if it's possible?

[leslie]

not sure if its relevant or not, but i am pretty confident that mike oldfeild uses dsp on his tubular bells 2 cd. He get some sort of pulsing effect,  how he does it is beyond my pay grade   https://www.youtube.com/watch?v=uDp-xpN7Xqg the dsp starts at 3.30 in i think. hope that helps

also in the movie, baby driver, the driver records and chops up conversations  to make music so its doable, but i dont know what software you would use

 

[matthere]

Thanks @leslie for your thoughts, sounds like some interesting effects on the video.

It’s the live aspect that I’m going to struggle with, but I may be able to work with just monitoring a chopped up and shortened version of the audio, without the need to slow anything down.. still trying to work out what might work for that.

[kye]

@IronFilm raises an important and fundamental issue - that if you're stretching time in any way you will get out of sync.  My method stayed in sync because it skipped past lots of content.

There is an element of time-stretching effect to what you're talking about, and there are algorithms that do such things, however fundamentally it means that the time it takes to listen to the output is different to the time it took to capture the input.

Have a think about how you're going to solve that and then we can talk about the details of implementing it  

[BTM_Pix]

Its doable with something custom written.

You record the incoming audio to files in increasing time chunks depending on the slow down percentage that you want, to take into account the time it will take to play them back.

So for example, lets take a half speed playback scenario.

The incoming audio (FILE 1) would be written to a file after the first 0.5 seconds and the playback of that file would begin immediately when its finished writing but at half speed.

Simultaneously, the still incoming audio would immediately start writing to a new file (FILE 2) which would need to be of sufficient length to cover the length of time the playback will take to complete the first file before it is ready to play this new file.

As our first file is 0.5 seconds, it will take 1 second to playback at half speed so our new file (FILE 2) will need to be 1 second long and this will immediately begin playing upon completion whereupon we begin recording another new file.

This file (FILE 2) will take 2 seconds to play back at half speed so our next file (FILE 3) will need to be 2 seconds long.

As this will take 4 seconds to complete playback at half speed our next file (FILE 4) will need to be 8 seconds long.

And so on and so on.

I've done you a spreadsheet illustration  

Depending on what you are going to run it on and how long the piece is you could do this in memory of course with buffers instead of writing to files.

The audio of course will be playing back at half pitch as well as half speed but there are enough affordable hardware solutions out there nowadays to take care of that.

[leslie]

40 minutes ago, BTM_Pix said:

Its doable with something custom written.

You record the incoming audio to files in increasing time chunks depending on the slow down percentage that you want to take into account the time it will take to play them back.

So for example, lets take a half speed example.

The incoming audio would be written to a file after the first 0.5 seconds and the playback of that file would begin immediately when its finished writing but at half speed.

The still incoming audio would immediately start writing to a new file (FILE 1) which would need to be of sufficient length to cover the length of time the playback will take to complete the first file before it is ready to play this new file.

As our first file is 0.5 seconds, it will take 1 second to playback at half speed so our new file (FILE 2) will need to be 1 second long and this will immediately begin playing upon completion whereupon we begin recording another new file.

This file (FILE 2) will take 2 seconds to play back at half speed so our next file (FILE 3) will need to be 2 seconds long.

As this will take 4 seconds to complete playback at half speed our next file (FILE 4) will need to be 8 seconds long.

And so on and so on.

I've done you a spreadsheet illustration  

Depending on what you are going to run it on and how long the piece is you could do this in memory of course with buffers instead of writing to files.

The audio of course will be playing back at half pitch as well as half speed but there are enough affordable hardware solutions out there nowadays to take care of that.

looks like you have another project btm ?

would you want to add any smoothing ? depending on the length of sound sample it it might get kind of choppy cutting it in half. just thinking out loud.

[BTM_Pix]

11 minutes ago, leslie said:

looks like you have another project btm ?

would you want to add any smoothing ? depending on the length of sound sample it it might get kind of choppy cutting it in half. just thinking out loud.

It very much depends on what you're developing it on but I'm guessing in the context of what this will be used for that some artefacts would be acceptable to achieve the real time aspect of it?

 

[matthere]

Thanks for the prompt responses everyone ?

As is always the way, in trying to explain this, it is making more sense slowly.. (I think?)

If it becomes possible to sample the live audio and monitor shortened audio samples, there probably isn't any need to slow the audio down through processing at the time of monitoring.

 As seen below if the live video (a) was recording at 96fps and played back at 24fps (b), then the live audio (d) would be chopped up and just the first 25% of the audio monitored (e), so that the person monitoring would hear the changes to the audio, then the live audio (d) could then be slowed down (c) and played along with the 24fps video (b).

 

[IronFilm]

On 10/11/2019 at 1:30 AM, kye said:

@IronFilm raises an important and fundamental issue - that if you're stretching time in any way you will get out of sync.  My method stayed in sync because it skipped past lots of content.

There is an element of time-stretching effect to what you're talking about, and there are algorithms that do such things, however fundamentally it means that the time it takes to listen to the output is different to the time it took to capture the input.

That very last partial segment of the sentence kye wrote at the end there is key, and is what I was referring to in my first post. 

Why even bother listening to it "live"? Because the MOMENT after you start to listen to it slowed down, then it is not live at all!

If you come up with a signal processing method which gives you even say a half second delay, which normally would be an annoyingly large delay to deal with when live, then I'd still regard this as an acceptable solution as you're not monitoring it truly live anyway. 

[matthere]

The idea that I was trying to explore/explain was around playing music at the same time as recording HFR video, I think I need to try some practical experiments, as I may have confused things by considering the slowing down of the audio, I am looking to monitor the audio "live" for a short time, then not monitor for a period of time, then monitor "live" again repeatedly so that I can be aware of the live audio (in a shortened "processed?" way) this would hopefully allow me the potential to sync camera movement, focus and zoom to the audio being recorded.

The chopping up and monitoring the live sound without slowing it down is what I am looking to explore, as I am not sure what software or hardware options might be available to explore this?

It should be said that this is still an idea in the development stage, so I may not be very clear in my explanations and may be off the mark in terms of what I need to complete the task, this is often the case with experimental art projects, apologies for the confusion this type of discussion may cause ?

[no_connection]

One option if you want interactive is having instant slomo button that will play back the last set of time in slow motion. Or trigger a recording that then gets played back.

As for audio a good place to start is Paul's Extreme Sound Stretch made to stretch audio out to extreme lengths (hence the name). Here linked in a portable form so you don't have to install anything. I'm pretty sure there would be some sort of version that exists with command line api to better suit your needs.

https://portableapps.com/apps/music_video/paul_stretch_portable

[KnightsFan]

On 10/15/2019 at 2:43 AM, matthere said:

I am looking to monitor the audio "live" for a short time, then not monitor for a period of time, then monitor "live" again repeatedly so that I can be aware of the live audio (in a shortened "processed?" way) this would hopefully allow me the potential to sync camera movement, focus and zoom to the audio being recorded.

What do you mean by "live?" Do you mean live as its recorded or live as its played? I suggest for clarification we refer to monitoring as its recorded or as it is played, it's less confusing that way (to me!).

Ultimately, you will need to do what @BTM_Pix suggested originally, which is record to a buffer, and selectively play out of that buffer. Essentially, record 4 seconds of audio to a file, slow that file down, and then play it over the next 16 seconds. You can continue to do this until you run out of file space.

At that point, you have two streams of audio: A. the one being recorded, and B. the one being played (which is always 4x the length of stream A). I assume the question isn't how to monitor either of those.

 

So I originally thought what you were trying to do is listen to the audio that you are recording, in a rough approximation of what it will sound like after it is slowed down. To do this, every 4 seconds, you listen to the 1st second slowed down to fill the whole 4 seconds. If this is the case, then all you need to do is introduce a third audio stream C, which is exactly the same as stream B, but only plays every 4th group of 4 seconds. This will monitor in real time as audio is recorded, with a slow down effect so you know what it will sound like later when it plays (although you won't be able to listen to all of it).

 

However, you drew a diagram which implies you are listening to normal speed audio, but only listening to 1 second every 4 seconds, the exact opposite of my original thought. So that would also work, but obviously you are listening to audio significantly after it was recorded. So that would bring me to this question: what does meaningful mean to you?

On 10/9/2019 at 7:06 PM, matthere said:

I am looking to find out if there might be a way to cut up a live audio signal and feed part of the signal in a way that it could be monitored in a meaningful way?

What are you trying to hear and adjust for in the audio?

 

 

On 10/15/2019 at 2:43 AM, matthere said:

The idea that I was trying to explore/explain was around playing music at the same time as recording HFR video...this would hopefully allow me the potential to sync camera movement, focus and zoom to the audio being recorded.

So to be clear, this is music being played on site, not a pre-recording of music? I ask because if both the video and audio are being recorded at the same time, and slowed in the same way, they should stay in sync already. So if a band is playing and you are shooting it with an HFR camera and audio, and you project the video in another room with that same audio, both slowed down, then they will be in sync.

[matthere]

@no_connection Thanks for the recommendation, Yes I use "Paul's stretch" software for other effects stuff and it works really well, but I am not sure how that could be used, other than post processing.

@KnightsFan Thanks for trying to make sense of the stuff bouncing around in my head ?

The idea I am trying to explore is around the creation of a rig that includes image and audio capture equipment, to record the process of live music production. (Think GH5, electronic synths, samplers and audio devices) and the monitoring that I am trying to produce is to help with the manipulation of the camera movement, framing, focus and zoom of the rigged camera. In a traditional slow motion set up, the prerecorded music is sped up to influence the fast action recording of motion of the artist to create synced footage and sound, in this situation I am considering slowing the treatment of the camera by slowing the audio played at the time that the footage is recorded.

So in reference to your question "live" refers to when the music is played.

You are correct in terms of the audio streams A being recorded "straight", audio B being 4x the length of A, and yes audio C would be for monitoring, and the "meaningful" element referred to, meaning the audio would allow me to adjust the camera in sync with audio events. So this seems to present the problem of trying to cut up the audio and pay back parts of the audio (a quarter of the audio seems to make sense if the frame rate is 4x the speed of the live action), this is to help the camera manipulation that would sync everything together.

I am grateful for folks perseverance, art ideas often contain counter-intuitive processes that often generate seemingly odd constraints, but these constrains can open interesting potential for future ideas. I am not sure if these experiments will produce meaningful and interesting outcomes, but part of the fun is the journey, and I appreciate everyone's willingness to participate in this ?

 

 

[KnightsFan]

9 hours ago, matthere said:

and the monitoring that I am trying to produce is to help with the manipulation of the camera movement, framing, focus and zoom of the rigged camera.

If you are slowing audio and video together from the same rig, you can manipulate the camera in real time based on the music being played by the band (stream A from my example), and all of those camera changes will already be in sync with the audio when they are slowed down together.

Am I getting this right? I'm probably missing something about what you're trying to accomplish still

 

Edit: Here's a more concrete example of what I'm saying. If you are recording the band, and you want to reframe when they change key, then you simply need to reframe when they change key. When the audio and video are slowed down, the slowed down video will still reframe exactly when the slowed down audio changes key.

Hopefully that will clarify how I am misunderstanding you.

 

 

9 hours ago, matthere said:

I am grateful for folks perseverance, art ideas often contain counter-intuitive processes that often generate seemingly odd constraints, but these constrains can open interesting potential for future ideas.

Experimenting is the fun part! It doesn't necessarily have to produce results at all.

[matthere]

1 hour ago, KnightsFan said:

If you are slowing audio and video together..

Am I getting this right? 

The intention is to have the audio at normal speed but in sync with the slowed down 96>24 fps video.

I am planning to create the audio alongside operating the camera, but in order to do this I am looking for a method of monitoring the audio stream C (which is audio as played live, but cut up and only listening to a quarter of the full audio duration).

"Experimenting is the fun part! It doesn't necessarily have to produce results at all."

I couldn't agree more ?

[KnightsFan]

1 hour ago, matthere said:

The intention is to have the audio at normal speed but in sync with the slowed down 96>24 fps video.

I am planning to create the audio alongside operating the camera, but in order to do this I am looking for a method of monitoring the audio stream C (which is audio as played live, but cut up and only listening to a quarter of the full audio duration).

Aha, I think I understand now. So the audio is going from the band out to the speakers without time manipulation? In that case, you would just monitor it normally.

As for syncing up, that requires you to know what the band is going to do, and choreograph beforehand. Example: Say you want to reframe when the key changes. Since your video is being slowed down, anything you film will be seen on screen at some point x seconds in the future. Therefore, you would have to reframe x seconds before the band shifts key.

I'm not sure what the purpose of chopping up the audio and playing back 1/4 of is anymore?