Thomas Huang

Gen AI & ML

From Concept to Animation & Video with Generative AI

As creative beings and artists, we're always pushing boundaries, seeking new ways to bring our visions to life. Generative AI, when guided by our artistic intuition, is fast becoming an invaluable tool in animation and video production.

I recently dove into how traditional creative workflows – from story scripts and thumbnails to storyboards – can blend seamlessly with multimodal AI to spark truly magical moments. My key takeaway? A solid grasp of art history, film language, artistic styles, and a clear personal vision are paramount to truly harnessing generative AI's power and directing its creative output.

Let me walk you through a creative process, from conceptualization to the final video outcome.

1. The Seed of an Idea: From Sketch to Story
My journey began with a sketch I'd held for a long time: a metaphorical visualization of a person reading a letter. I wanted to depict profound affection and imagination overflowing like an abundance of water, teeming with whimsical fish and jellyfish. This was the spark!

2. Art Direction: Cultivating a Vision and Style
I dreamt of a Magical Realism story, marrying the intricate, symbolic patterns of Gustav Klimt and the elegance of Art Nouveau with elements of photorealism and an illustrative quality. My goal was to infuse the scene with rich symbolism and a dreamlike ambiance.

3. Look Development with Generative AI

Using generative AI models like Flux-Kontext and Gemini Nano-Banana, I initiated an iterative dance. I provided multimodal inputs, starting with my original sketch alongside detailed text descriptions of the environment, theme, action, movement, and ambiance. This back-and-forth process allowed me to feel both in control of the direction and inspired by the AI's surprising capabilities.

Here are some of the look development images that emerged from this collaboration:

4. Image-to-Video Generation
With my desired style frame locked, I moved to an Image-to-Video model like WAN. I provided this key frame along with a precise description to guide the animation, specifying elements like:

Camera: [e.g., "Steady mid-shot, slight dolly forward, underwater perspective"]
Action: [e.g., "Person gracefully "swimming" through the water-filled hallway, letter held carefully, fish and jellyfish swirling around them."]
Mood: [e.g., "Dreamy, wondrous, magical, gentle movement of water and creatures"]
Lighting: [e.g., "Subtle light rays filtering from above, casting dappled patterns on the floor"]

This involved another round of experimentation and prompt refining until the motion and narrative flowed just right.

This has been a quick, thrilling experiment, powerfully illustrating how AI can become an extension of an artist's and filmmaker's creative toolkit. It allowed me to maintain my directorial voice while evolving my process and enhancing the final output.

While longer-form productions present different challenges (such as ensuring character consistency, directable movements, realistic physics, and precise camera control), this exploration beautifully highlights the potential for AI to unlock new creative avenues – always provided that the AI model is sourced and trained responsibly.

Report abuse