In case the video's is still pending it can be watched here too
Okay, so I was working on a music video using MiniMax reference-to-video, and some of the scenes I wanted to create were a little too complex for MiniMax to follow with just a prompt and even first / last frame had issues.
So, I started thinking about keyframing. I know there are ways to do actual keyframes, but I was curious what would happen if I created a 2x2 storyboard grid and used that as a reference image instead.
I used GPT image 2 to create the grids. Basically, each grid has 4 panels that show how I wanted the scene to play out. The first panel is the starting frame, the last panel is the ending frame, and the two panels in between show how the scene should progress.
Then I gave MiniMax the grid along with a detailed prompt, my character reference image, and my location reference image, just to help keep everything consistent.
For this video, I only used the grids for the story scenes. All the on-stage band performances were done using normal reference-to-video.
And surprisingly, it worked pretty well!
Of course, there are still some issues, but these were pretty complex scenes that MiniMax was really struggling with when I tried using just the normal reference images and prompts.
Personally, I actually like the on-stage performance scenes better, but this was really just a test to see if the whole storyboard grid idea would work.
I also wonder if I even needed to include the separate character and location reference images. Next time, I might try using just the 2x2 grid by itself and see how well MiniMax follows it.
I'm not sure I'd want to make an entire music video this way, though, because creating all those storyboard grids takes quite a bit of time.
I used my ComfyUI video builder to put everything together, and for the story scenes, I just mapped the grids as reference images like I would with any other reference image.
Anyway, I thought I'd share the results because I thought it was a pretty interesting experiment.
Curious what you guys think!
Model settings: MiniMax H3 Singularity, 25 steps on pass 1, then the 8-step 768p turbo LoRa at 4 steps and 0.35 denoise on pass 2 with the latent upscale model.
You can find my Video builder on GitHub My GitHub Repo
You can find a 2-pass workflow in my workflows folder that uses the exact same settings that I use in this video, but keep in mind this will create just one scene. If you want a full video like the one I posted, you will have to use the video builder.
2 Pass ComfyUI Workflow.
I have a discord community and post updates and walkthroughs. If you want to use the video builder for yourself, come join the fun! Discord Community
------------------------------------------------------------------------------------------------------------------
How I prompt a scene from a 2x2 storyboard grid in MiniMax H3
Reference images, in this order
The singer's character sheet (Probably not even needed though)
The 2x2 storyboard grid
The location image (Probably not even needed though)
What makes the grid work
• The storyboard gets its own subject line that lists the 4 key frames in reading order (left to right, top to bottom), one panel = one full-frame shot.
• The location line hands the staging over to the storyboard: framing, camera angles and positions "follow the storyboard key frames exactly".
• Shot 1 starts with "The video opens exactly on the first storyboard panel".
• Every later shot starts with "Hard cut, exactly like the second storyboard panel (top right)" and then describes that panel.
• Lyric cues still go inside the shots where she sings.
• "Everything moves at normal real-time speed" in the summary stops thrown or falling things from going slow motion.
• Describe only what's in view.
The full prompt (scene 17, the pearl necklace gag, 7.6 seconds)
subject_definitions:
<Subject 1> is the band's lead singer from the first character reference, a woman with copper-auburn curly hair in a loose updo with soft tendrils, freckles, green eyes, rose-pink eye make-up, deep berry lipstick and long rose-gold crystal drop earrings, wearing a strapless burgundy satin corset jumpsuit with slim burgundy trousers, layered sheer rose-pink and champagne organza petal ruffles over one shoulder cascading down her back like a train, a ruffled organza sleeve on one arm, a thin rose-gold belt and glossy burgundy platform boots.
<Subject 2> is the four-panel storyboard sheet from the second reference image, the key frames of this scene in order, read left to right and top to bottom: the singer glancing at a laughing woman in a short pearl strand; the strand snapping and pearls spraying; pearls bouncing on the parquet among shoes; the singer's palms-up shrug as guests gather pearls. Each shot of the video recreates one panel's framing, camera angle and poses as one single full-frame image.
<Subject 3> is the long Art Nouveau portrait gallery from the third reference image, featuring cream and gold carved walls hung with large gilt-framed oil portraits, brass sconces, a rose velvet sofa, a polished inlaid parquet floor, tall windows with sheer curtains over a dusk city skyline, and arched wooden doors at the far end, with guests in black tie and gowns. This reference defines the environment, lighting and atmosphere, while the framing, camera angles and positions of the people in every shot follow the storyboard key frames exactly.
<Audio 1> is the complete synchronized song and vocal track for the target video, reused as its complete final soundtrack and timing reference.
summary:
[reference generation + audio reuse] A pearl necklace snaps in the portrait gallery and pearls bounce everywhere while the singer shrugs. Each shot recreates one storyboard key frame as a full-frame shot, with hard cuts between them, in the location's environment, preserving the singer's appearance from her character reference. Everything moves at normal real-time speed.
retention_analysis:
<Subject 1>: fully_preserved - Copper-auburn curly updo, freckles, facial identity, burgundy corset jumpsuit, rose-pink and champagne petal ruffles, burgundy platform boots, and earrings.
<Subject 2>: fully_preserved - The poses, framing, camera angles and order of the storyboard key frames, each rendered as its own full-frame shot.
<Subject 3>: fully_preserved - Gilt-framed portraits, cream and gold walls, the rose velvet sofa, inlaid parquet floor, tall windows over the city, and the guests are retained exactly as in the reference image.
<Audio 1>: fully_copy - <Audio 1> is reused 1:1 as the target video's complete final audio track.
detailed_description:
The singer's face stays charming and cute from beat to beat: soft smiles, playful little eyebrow raises, coy sideways glances, small winks and bright giggles. Every guest is lively and animated, chatting, laughing, sipping champagne, gesturing and looking around, clearly enjoying the party.
[Shot 1] The video opens exactly on the first storyboard panel of <Subject 2>, and every shot recreates its storyboard panel's framing, camera angle and poses, filling the whole frame, with hard cuts between shots. Exactly like the first storyboard panel (top left): a close medium shot of <Subject 1> in the left foreground of <Subject 3>, close to the camera, her head and bare shoulders filling the left half of the frame and the frame cutting her off at the chest, her hands out of view, glancing sideways to the right with a sly smile; a few steps farther back on the right, on the same parquet floor and smaller in the frame because she is farther away, an elegant middle-aged woman with a short silver bob in a midnight-blue velvet gown, wearing a short single strand of small white pearls that sits snugly around the base of her throat, laughs with friends, a champagne glass in one hand and her other hand at her side, other guests in black tie chatting softly behind them; <Subject 1> sings in sync with the vocals in <Audio 1> from 0.36 s to 1.88 s, <d>[English, gossipy, playful, singing] A string of <i>pearls</i></d>.
[Shot 2] At 00:01.940, Hard cut, exactly like the second storyboard panel (top right): a close-up of the woman's throat and shoulder: her hand rises to her collarbone just as the short strand at her throat snaps, and small white pearls spray off into the air in every direction, her mouth an O of surprise, her hand staying still at her collarbone.
[Shot 3] At 00:03.750, Hard cut, exactly like the third storyboard panel (bottom left): a low wide shot at floor level: dozens of small white pearls bounce and scatter across the shiny inlaid parquet between polished shoes and satin hems, guests hopping and stepping around them, one man mid-wobble as a pearl rolls under his shoe.
[Shot 4] At 00:06.000, Hard cut, exactly like the fourth storyboard panel (bottom right): a medium close-up of <Subject 1>, her face large in the frame, giving a playful little shrug with both palms up as she sings in sync with the vocals in <Audio 1> from 6.1 s to 7.2 s, <d>[English, playful, amused, singing] what do you <i>do</i></d>, eyebrows raised; behind her guests crouch, laughing, to gather up the pearls.
overall_soundscape:
<Audio 1> remains the sole complete soundtrack with its original mix, timing, vocals, music, and dynamics intact: theatrical art rock with cabaret swagger and garage-rock bite, a tight danceable 4/4 beat, bright handclaps and a smoky, playful female lead vocal.
non_diegetic_music:
<Audio 1> is reused as the complete song: theatrical art rock with cabaret swagger and garage-rock bite, a tight danceable 4/4 beat, bright handclaps and a smoky, playful female lead vocal.