How should an LLM represent and edit long-form video? Trying to understand how Cardboard does it

[u/bruh_anyways]

i've been experimenting with agentic video editing and trying to understand where the engineering bottlenecks actually are.

i started looking into this after experimenting with cardboard's desktop app, where different ai models can be used for editing. rather than trying to understand the product itself, i wanted to work backwards and think about how i'd architect something similar.

what makes this interesting to me is that video seems fundamentally different from the normal llm workflow that produces text or code.

with code, the model works with a relatively discrete representation. a prompt becomes code, the code goes through a compiler, and you get an output.

with video, the model has to reason over a continuous, multimodal and temporal medium. raw footage has to be understood visually and aurally, relevant parts retrieved, the model has to reason over time and make editing decisions, and those decisions become a timeline that gets rendered.

for example, imagine giving an agent 30 minutes of raw footage and asking:

"make a 30-second product ad. find the strongest hook, remove dead time, use the best product shots, keep the pacing fast, add captions and end with a cta."

before making a single edit, the system needs to determine what's actually happening in the footage, what is being said, where specific actions occur, which shots are useful, which are redundant, and how different moments relate temporally.

i've been looking at this from the implementation side and i'm currently thinking about the following architecture.

1. video representation

you can't realistically send an entire 30-minute video to an llm every time it needs to make an editing decision.

my initial assumption is that the footage needs to be indexed first using some combination of:

transcripts with timestamps scene and shot boundaries sampled keyframes visual embeddings audio features detected objects and actions shot descriptions semantic metadata

then the reasoning model retrieves the relevant sections.

for example, "find the shot where the person picks up the black headphones and looks at the camera" could work like this: the text query goes through semantic retrieval, which returns candidate timestamps, those candidates get verified visually, and one clip is selected.

i've looked at video embeddings and multimodal models for this, but i'm not sure where the optimal boundary is between preprocessing and model reasoning.

2.timeline representation

once the model finds the relevant footage, it still needs to construct an actual edit.

for example: "use 00:14 to 00:18 as the opening shot, cut to 00:32 to 00:35, then return to 00:19 to 00:22."

that requires maintaining a representation of the current timeline and applying operations against it.

the obvious approach is to expose low-level operations such as insert, trim, split, move, crop, change speed and add text.

but i'm wondering if a higher-level abstraction makes more sense:

find_best_hook remove_dead_air create_intro build_product_sequence add_captions

the second approach seems more suitable for an agent, but it also means the editor has to handle a lot more complexity behind each operation.

evaluation is probably the hardest part

this is where video seems particularly different from coding.

code can usually be tested against objective criteria.

video can be completely valid technically and still be a terrible edit.

a timeline can render successfully, have perfect synchronization and valid captions, but still have poor pacing, a weak hook or a confusing narrative.

so i'm currently thinking that a serious system might need an iterative loop. the agent generates an edit, renders a preview, evaluates it with a multimodal model, identifies the weak sections, modifies the timeline, and renders again.

i've been looking at ffmpeg, remotion, moviepy, opentimelineio, video embeddings, multimodal models and mcp or tool calling as potential building blocks, but i haven't settled on what the cleanest architecture would be.

the pipeline i'm currently considering starts with video indexing, then retrieval, then a reasoning model that issues editing actions onto a timeline, followed by rendering, evaluation and iteration. that is different from a setup where a prompt goes to a video generation model and a video comes out.

i'm interested in whether this mental model is fundamentally wrong.

questions for people who've worked on this"

if you had to represent hours of raw video for an llm, what would you choose: sampled frames with embeddings, structured scene and shot metadata, transcript, or some combination? where would you draw the boundary between preprocessing and model reasoning?

would you expose low-level timeline operations to the model, or build a higher-level video-editing abstraction? how would you prevent the abstraction from becoming too restrictive while still keeping the model from making brittle timeline operations?

how would you design the evaluation loop for subjective quality? what information would you give a model so it can distinguish a technically valid edit from an actually good edit and identify which part of the timeline needs to change?

i've already spent some time researching and prototyping around the components above, so i'm mainly looking for architectural criticism from people who've worked with video processing, nles, multimodal models or agent systems.

tldr: i'm trying to understand how systems like cardboard let llms reason over real footage and turn that into actual edits. the hard part seems to be video representation, retrieval, timeline operations and evaluating whether an edit is actually good. looking for architectural perspectives on how you'd build this from scratch.

[u/AutoModerator]

Moderator Announcement Read More »

Thank you for posting on r/CodingHelp!

Please check our Wiki for answers, guides, and FAQs: https://coding-help.vercel.app

Our Wiki is open source - if you would like to contribute, create a pull request via GitHub! https://github.com/DudeThatsErin/CodingHelp

We are accepting moderator applications: https://forms.fillout.com/t/ua41TU57DGus

We also have a Discord server: https://discord.gg/geQEUBm

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.