Workflow for editing large projects?

[kye]

I am just starting to edit the largest project I've taken on, and I can quickly see myself getting a bit lost in the footage if I'm not careful.

It's from a 5 week trip I took through Italy and there are ~5400 source clips totalling almost 6 hours (average 4s per shot).  I have done a leisurely (~10 hour) first pass and pulled in any sections that I think might be useful in the edit, and I'm down to 2300 shots and just over an hour.  My strategy with this first pass was to cull the dud stuff and never go back to the source footage again, only ever looking at this timeline for the rest of the edit.  I typically do each pass on the footage and then duplicate the timeline to work on the next pass, so I can easily refer back if there are shots I culled previously but want to find again, etc.  I like creating little sequences that tell micro-stories with 3-5 shots and linking them together to create the broader story arcs, so I find I often choose hero shots then need the other random shots to make the story work, so I refer back quite a bit for the supporting footage.

My next step is to look at what sequences and locations I have so I can get a high-level view on the project, ahead of deciding the overall structure, which locations and sequences I will keep, etc.  I'm doing this with markers at the moment, but will probably make a list in a text editor outside the NLE.  I don't have a target format or final duration so for me it's just about shaping it the way I want it, then paring it down so it's constantly interesting, and it will be however long it ends up.

My challenge will be that my footage isn't quite in the order I want it to be in, and I'll need to go through a process of rearranging it before I get into it on a shot-by-shot basis, and I fear that I'll make a mess of it and confuse and frustrate myself in this phase.

My understanding is that people who edit docos do this rearranging all the time, essentially making the story in the edit.  What's the best way to manage this volume of footage during this process?  Is it just an admin process, do people sort footage using Compound Clips, separate Timelines, Bins, colour coding the clips, something else?

I'm using Resolve, but I expect that the tech doesn't really matter, it's more of an admin challenge of sorts.

Who is editing on this scale?  What do you do?  What advice do you have for me?  

[thephoenix]

that is the moment you find all these little things like labels, naming, metadata (and you name it) are getting so usefull i guess.

never did long project but always been curious how they do it.

sure it's hours if not days of organizing things right.

i am sure you have good tutorials on youtube

[kye]

2 hours ago, thephoenix said:

that is the moment you find all these little things like labels, naming, metadata (and you name it) are getting so usefull i guess.

never did long project but always been curious how they do it.

sure it's hours if not days of organizing things right.

i am sure you have good tutorials on youtube

It's interesting but this is one of the things that you don't really get people on YT talking about, it's kind of a blind spot.  I guess that people who make YT videos, therefore have short and frequent production cycles, aren't experts in managing long infrequent production cycles

Kind of like most things in high-end professions, the people who know how to do it are either off doing it, or are selling the content and not giving it away for free! ???

[BTM_Pix]

1 hour ago, kye said:

It's interesting but this is one of the things that you don't really get people on YT talking about, it's kind of a blind spot.  I guess that people who make YT videos, therefore have short and frequent production cycles, aren't experts in managing long infrequent production cycles

Kind of like most things in high-end professions, the people who know how to do it are either off doing it, or are selling the content and not giving it away for free! ???

You should probably take a look at This Guy Edits channel on YouTube.

This video is possibly not nuts and bolts enough for what you are after but the concepts are there.

The more in depth stuff is accessed via his Patreon account which is fair enough as the stuff he has for free to view on his YouTube channel is always very good.

 

 

[mkabi]

I usually start sorting through the mess...

Day 1, Day 2, Day 3, etc.

Then I start sorting each day by theme, if similar themes arise, I group them together rather than by days.

So multiple days become a few select themes... then each theme... I usually sort as the Good, the bad and the ugly.

The Good shots will definitely be in the edit.

The Bad is there only if you have no resort, and you need something essential for the story.

The Ugly is for the recycling bin or blooper reel.

 

[hk908]

I think the newest version of Premiere has a "freeform view" mode or similar where you can group shots loosely to get a feel for what flows right. Could try that out, esp if you've already culled a bit of the source clips. I've never used it but it seems interesting.

 

https://www.youtube.com/watch?v=v5iguTGlDcc

[User]

Seriously great post Kai... thank you!

I'm deep into the edit on a sprawling doc film set in South Asia with close to a 1000 pages of translations and 17,000 clips I lensed and hundreds of other associated files. 4 main characters and 3 minor characters all with their own trajectories... and besides you guys, I'm entirely alone on this. It's been an absolutely massive undertaking.

19 hours ago, kye said:

My strategy with this first pass was to cull the dud stuff and never go back to the source footage again, only ever looking at this timeline for the rest of the edit.  I typically do each pass on the footage and then duplicate the timeline to work on the next pass, so I can easily refer back if there are shots I culled previously but want to find again, etc.  I like creating little sequences that tell micro-stories with 3-5 shots and linking them together to create the broader story arcs, so I find I often choose hero shots then need the other random shots to make the story work, so I refer back quite a bit for the supporting footage.

This is exactly how I'm working... so it's reassuring to hear that this is your approach too.

For this doc film, I've stumbled upon a method where I place everything on timelines and work my way through them cutting out the fat and then building scenes as my characters get caught up in something interesting.

I also colour code interesting scenes and shots so I can easily find them later and then will move towards organizing them in the months ahead.

Some years ago I spent months key wording much of the project only to realize that, although this can be helpful to quickly find stuff, it might have actually been better to just lay stuff out on the timeline and work through it and build.

It may not apply to you in the same way since it doesn't sound like you are following protagonists over longer periods but laying out each characters narrative trajectory is something that I'm going to be facing very soon. My approach will be to find the most powerful (emotional) moments for all of the characters and place them on the timeline at (what might be) the climactic 85% part of the timeline and then build everything towards these moments. This is really the nuts and bolts of story... I know from attending some festival masterclass talks that this stuff can be talked about in a broad sense, but the real heavy lifting takes months (if not years) into a doc film between the director, editor, story consultant and producer.

@BTM_Pix Thanks for the links, I founds these a few months ago... there is really so very little out there (blind spot as Kai says) on the web about how to do this stuff.

@hk908 I tried the Freeform View and found it of little use... but it might prove helpful down the road... I sort of doubt it.

19 hours ago, kye said:

My next step is to look at what sequences and locations I have so I can get a high-level view on the project, ahead of deciding the overall structure, which locations and sequences I will keep, etc.  I'm doing this with markers at the moment, but will probably make a list in a text editor outside the NLE.  I don't have a target format or final duration so for me it's just about shaping it the way I want it, then paring it down so it's constantly interesting, and it will be however long it ends up.

I like what you've said about maintaining a 'high-level' view over the project... all to often it can be daunting to see the forrest through the trees so one really has to step back and try to envision where things can and should go. Notice the coincidences and trust that, if you are keeping a level head, all will be revealed as you work into things. Trust the process Kai and at the same time try and include people you can trust to lend their insights, even if they don't work on these types of projects.

And finally, if it give you goose bumps... you are on the right track!

[kye]

@BTM_Pix thanks for those videos, I haven't seen them.  I am a subscriber to This Guy Edits and it's great stuff, possibly the only video editing YT channel.

@mkabi I like the Good / Bad / Ugly analogy, it works well.  

@User 17000 clips would be a nightmare for most I'd say.  I'd also imagine that lots of those clips would be interviews where it's not one moment, more like an entire dump trucks worth of dirt in a pile and you have to dig through and really hope there are some gold nuggets in there somewhere.

I'll have to go through my YT watch history and browser history and find some of the good editing stuff I've watched over the last few years.  There's quite a bit, unfortunately it's not nearly enough, especially compared to how many videos are telling you to buy an ND filter and use a 180 shutter.

As my videos are basically just random shots of where we were, what we did, and that sort of thing, for me it's not about re-arranging the footage hugely, as the plot is mostly written, the story elements (inciting incidents, challenges, etc) are all from what happened, so my job is basically filtering all that down into a highlight reel of sorts.  My films don't really have much dialogue either, so they're mostly visual storytelling with music and ambient sounds.

To that end, my first pass is an assembly where I pull every potentially usable bit onto a timeline to cull the Ugly as Mkabi says.  Then second pass is to go through that and on a shot by shot basis I sort them into several categories using tracks.

Track 1 is the default track, if a clip is of no interest it stays here
Track 2 is nice B-roll that might be useful for establishing shots or whatever
Track 3 is spectacular B-roll, potentially being visually stunning or if I know it's a critical shot for something we did
Track 4 is any usable shot of the people I was with (because people are more important than random b-roll, however pretty it is)
Track 5 is any great shots of people I was with (beautiful, funny, memorable, etc)

This stays as the first timeline.  Then I duplicate that timeline and delete all the shots from tracks 1 and 2.  Then I cut out entire sections I don't care about.  Then I go through and start making the little micro-stories by only putting back enough shots from Track 2 (or track 1) to make the great shots make sense.  At this point I can get a sense of how many people shots I have and start removing the bad ones if I have too many of a person.  For me, coverage of the people is more important than the quality of the shot, so if I have a usable but not-great shot of a person then I'll probably leave it in rather than them not really appearing much (or at all) in the video.

Then comes music, and I start editing the timing of clips because now I can get a sense of pace and rhythm from the music.  Then it's titles and grading etc.

I've just started using Markers in Resolve and it's quite good actually.  I have the list of markers open in the side panel, and it kind of treats them like chapter headings, so if you have marker 1 on clip 10 and marker 2 on clip 20, then if the play head is anywhere between the two markers it keeps marker 1 highlighted, so you kind of always know where you are.

The problem I've had with this edit is clip order.  This trip was shot with the XC10 and my iPhone.  Ideally I'd sort the footage by timecode, but the iPhone doesn't have it, so you get those in the wrong order (all at the start IIRC) so I sort by Date Created, which mostly works.  The issue is that as I'm editing on the train from my laptop, I've generated proxy media and the dates on the proxy media isn't quite right because (for whatever reason) Resolve didn't process the clips quite in sequential order so the date created for the proxy files is a bit messed up, and you get different events split into 2 or 3 chunks.  
By creating Markers every time there's an event change in the timeline I can detect separate chunks of clips and easily highlight all the clips from that chunk and move them next to the other chuck of clips from the same event, thus giving me that next level up view and sorting the footage properly.  Then once all the footage from an event is together I can just zoom in and work out what shots to keep and what stories to tell, or even to mention that event at all.

So far so good.

[User]

1 hour ago, kye said:

and you have to dig through and really hope there are some gold.

Oh I don't have to hope... I've got an explosive motherfucker that is going to get me blacklisted from my favourite country for all the right reasons  But yes the nature of what I do is verite and so much of this depends on what happens spontaneously... and that can mean long rolls of footage.

Thanks again for taking the time to lay out your process.

[mkabi]

10 hours ago, kye said:

 

@mkabi I like the Good / Bad / Ugly analogy, it works well.  

Just for clarity, I go into each "themed" folder and create 3 more folders, "The Good"; "The Bad" and "The Ugly"

For Gen Z people that don't get it, its an Eastwood reference. 

[Xavier Plagaro Mussard]

YouTube if full of travel videos. Only a long bad travel video can be worse than a short bad travel video. I see a lot of 2-3-4 minutes long videos that have zero continuity and are impossible to watch completely. Some randoms clips with background music. 

I would warmly recommend that you do a version for yourself/friends/family and a shorter universal version, to put online. 

[kye]

6 hours ago, Xavier Plágaro Mussard said:

YouTube if full of travel videos. Only a long bad travel video can be worse than a short bad travel video. I see a lot of 2-3-4 minutes long videos that have zero continuity and are impossible to watch completely. Some randoms clips with background music. 

I would warmly recommend that you do a version for yourself/friends/family and a shorter universal version, to put online. 

I agree that a great number of travel videos aren't well made at all.  I typically end up with videos in the 3-6 minute range, but these are for friends/family only.  Making a video that is nice to watch for strangers is a completely different challenge and I'm not sure I'd be much good at it TBH.

I have thought that I could take some of the more interesting trips I've had and make a 'public' version, but I haven't gotten that far yet.  In terms of longer travel videos, I really like Kraig Adams on YT who seems to have pioneered the Super-Vlog genre of long form travel videos.  His Japan one is particularly good, and he has also made some BTS videos too which are quite informative.

[webrunner5]

Most of the travel videos from people I follow are in the 12 to 15 minute range. I am pretty impressed that these people can come up with interesting content ranging form 2 to 3 a week to one a week.  It has to take one hell of a lot work to pull it off. And on average I see very few noticeable flaws in them.

Most are using a Canon with DPAF DSLR, one uses a GH5, one is doing all the RX100 stuff and the RX0 mk II. And the usual suspects of a GoPro, DJI drone. I suspect the new Osmo with replace a few GoPros soon.

[fuzzynormal]

There's no "one trick" to making this efficient right out of the gate. It's esentially the same thing as learning a language or learning the guitar. 

You practice and you learn techniques that allow you to succeed.  Those techniques and that style depends heavily on what kind of person you are.

If you're good, you hone a sense for grabbing your "selects" and it gets you to the end-game faster, but you have to develop that muscle. 

FWIW, your project is big, but still modest compared to what typically gets shot for a doc. I hate to think what my footage-to-final ratio usually is...100:1'ish?

11 hours ago, mkabi said:

its an Eastwood reference. 

I'd rather that offered as a Leone reference. 

Man, loved the way that guy did things. 

[kye]

13 hours ago, webrunner5 said:

Most of the travel videos from people I follow are in the 12 to 15 minute range. I am pretty impressed that these people can come up with interesting content ranging form 2 to 3 a week to one a week.  It has to take one hell of a lot work to pull it off. And on average I see very few noticeable flaws in them.

Most are using a Canon with DPAF DSLR, one uses a GH5, one is doing all the RX100 stuff and the RX0 mk II. And the usual suspects of a GoPro, DJI drone. I suspect the new Osmo with replace a few GoPros soon.

I agree that Osmo Action will be a hit amongst YT creators quite widely, I'd love for it to be what I was looking for!

The people who do this stuff regularly are basically shooting and editing machines.  I see the good YT creators eventually talking about burn-out, creative stagnation and ruts, and taking time off followed by "where I've been" videos upon return.  I think the pace is unforgiving well beyond what I can imagine.  I remember Casey Neistat saying that he spent 4-9 hours editing each daily vlog, and he did that uninterrupted for something like 500 days.  It's almost a good thing that social media is addictive or we'd all think of them as masochists!

One lesson we can take from this whole thing though is that practice really helps.

11 hours ago, fuzzynormal said:

There's no "one trick" to making this efficient right out of the gate. It's esentially the same thing as learning a language or learning the guitar. 

You practice and you learn techniques that allow you to succeed.  Those techniques and that style depends heavily on what kind of person you are.

If you're good, you hone a sense for grabbing your "selects" and it gets you to the end-game faster, but you have to develop that muscle. 

FWIW, your project is big, but still modest compared to what typically gets shot for a doc. I hate to think what my footage-to-final ratio usually is...100:1'ish?

I know my project isn't big by other peoples standards.  It was just bigger than I could deal with using my existing knowledge and approach, thus the request for advice.  It's always good to stretch yourself and keep learning

I've heard of docs commonly being 100:1 and blockbuster movies sometimes go well beyond that IIRC, I remember a ratio above 400:1 which is just madness!  Plus with a doc you don't know where the story is and so until you work out what the story (or stories) are you can't even work out where the 'good bits' are.  At least in my situation, as well as a scripted piece, you can review the footage and have a general idea about what you can cull on face value.  Docos are only a couple of stops short of being given a newspaper and having to write poetry by cutting it up into individual words and rearranging them!

[User]

4 hours ago, kye said:

Docos are only a couple of stops short of being given a newspaper and having to write poetry by cutting it up into individual words and rearranging them!

Exactly. To me, docs are one of - if not the most - challenging genres to work in. Especially ones that drift across cultures and languages.

[Novim]

@kye Just a suggestion: find the clips that could be organized in repetition (like leit-motifs), it could help to the structure of the whole. Another suggestion: since in every artwork the meaningful end comes usually before the actual last element, find the end beforehand (I tend to have 2-3 endings, as a kind of a coda).

[kye]

1 hour ago, Novim said:

@kye Just a suggestion: find the clips that could be organized in repetition (like leit-motifs), it could help to the structure of the whole. Another suggestion: since in every artwork the meaningful end comes usually before the actual last element, find the end beforehand (I tend to have 2-3 endings, as a kind of a coda).

Interesting ideas.  I'm familiar with repetition and evolution in music, but how would you do this kind of thing visually?

Just thinking out loud here, but at a shot-level it would be kind of hard to create in the edit, but you could have little patterns in sequences like a wide-medium-tight at the beginning of each scene perhaps, or a certain rhythm in the cuts perhaps.  I heard that in film school they talk a lot about layers, so you have these recurring themes that don't directly influence the plot but kind of weave the footage together on an aesthetic sense perhaps, like colour schemes, or set design, or language, etc.  I guess the answer is probably all of these things.

In a sense I have developed my own visual language with a title sequence that is similar for all videos, and I tend to do straight cuts when I'm cutting within a location or event, and use dissolves when I'm transitioning from one location or event to another one.  I have noticed that other travel films have used a quick fade-to-black for scene changes which gives a different feel.  I'm still working out how best to get from one location to another, if it should be travel b-roll or not, etc, but it is a repetition of sorts.

I mostly just edit down the events I film in the order they happened, and I know that docos are often highly chopped up.  It made sense to me how you could have a three-act structure (or other structure) in a drama, and also in a doco, but I couldn't see a link in-between my sequential editing and these structures, but I think I'm starting to understand how I could subtly weave such things into my edits.  It wouldn't be obvious, and I already have a kind of emotional arc by using the shape of the music I include, but I can see that you could push things further with subtle things like edit timings, grading changes to make the film ebb and flow a bit more, etc.  These techniques might be the way to give my films an ending without deviating from the chronology of how things actually occurred, which I'm keen to stick to, at least for the moment.

Definitely something to contemplate!

[Novim]

We are on the similar line of thinking, it seems to me (subtle, almost "unnoticable", more unconscious-driven repetitions of little patterns). And, after you finish, let it remain for a day or two (or more, better), and than come back and give to it the "final cut". Your task is to find a shape to a mass of clips, to arrange a chaos, and also you could use, perhaps, the school  so-called "3 minutes" rule (i.e. after every 3 minutes sequence put something diffrent, or begin a new sequence). - All I say is "theoretical", of course, and just suggestions, but sometimes it helps. ? Good luck!

(I also use, very often, XC10 for docos, now combined with DJI Osmo Pocket which is better than my 6+ iPhone because of stabilisation and 4K. P4K is when I want to go special. ? )

[joema]

On 5/28/2019 at 6:54 PM, kye said:

I am just starting to edit the largest project I've taken on, and I can quickly see myself getting a bit lost in the footage if I'm not careful...~5400 source clips totalling almost 6 hours (average 4s per shot).  I have done a leisurely (~10 hour) first pass and pulled in any sections that I think might be useful in the edit, and I'm down to 2300 shots and just over an hour.  My strategy with this first pass was to cull the dud stuff and never go back to the source footage again, only ever looking at this timeline for the rest of the edit....My challenge will be that my footage isn't quite in the order I want it to be in, and I'll need to go through a process of rearranging it before I get into it on a shot-by-shot basis, and I fear that I'll make a mess of it and confuse and frustrate myself in this phase...My understanding is that people who edit docos do this rearranging all the time, essentially making the story in the edit.  What's the best way to manage this volume of footage during this process?  Is it just an admin process, do people sort footage using Compound Clips, separate Timelines, Bins, colour coding the clips, something else?....I'm using Resolve, but I expect that the tech doesn't really matter, it's more of an admin challenge of sorts....Who is editing on this scale?  What do you do?

I worked on a collaborative team editing a large documentary consisting of 8,500 4k H264 clips, 220 camera hours, and 20 terabytes. It included about 120 multi-camera interviews. The final product was 22 min.

In this case we used FCPX which has extensive database features such as range-based (vs. clip-based) keywording and rating. Before touching a timeline, there was a heavy organizational phase where a consistent keyword dictionary and rating criteria was devised and proxy-only media distributed among several geographically distributed assistant editors. All multicam material and content with external audio was first synchronized. FCPX was used to apply range-based keywords and ratings. The ratings included rejecting all unusable or low-quality material which FCPX thereafter suppresses from display. We used XML files including the 3rd-party utility MergeX to interchange library metadata for the assigned media: http://www.merge.software

Before timeline editing started, by these methods the material was culled down to a more manageable size with all content organized by a consistent keyword system. The material was shot at 12 different locations over two years so it was crucial to thoroughly organize the content before starting the timeline editing phase.

Once the timeline phase began, a preliminary brief demo version was produced to evaluate overall concept and feel. This worked out well and the final version was a more fleshed out version of the demo version.

It is true that in documentary, the true story is often discovered during the editorial process. However during preproduction planning there should be some idea of possible story directions otherwise you can't shoot for proper coverage, and the production phase is inefficient. 

Before using FCPX I edited large documentaries using Premiere Pro CS6, and used an Excel Spreadsheet to keep track of clips and metadata. Editor Walter Murch has described using a Filemaker Pro database for this purpose. There are 3rd party media asset managers such as CatDV: http://www.squarebox.com/products/desktop/ and KeyFlow Pro: http://www.keyflowpro.com

Kyno is a simpler screening and media management app which you could use as a front end, esp for NLEs that don't have good built-in organizing features: https://lesspain.software/kyno/

However it is not always necessary to use spreadsheets, databases or other tools. In the above-mentioned video about "Process of a Pro Editor", he just uses Avid's bin system and a bunch of small timelines. That was an excellent video, thanks to BTM_Pix for posting that.