Where AI is Useful for Video Production and Where It Falls Flat

The world has been ablaze with different AI models and their capabilities since the release of ChatGPT. While AI video generation is not yet on the same level as image generation, it is rapidly advancing and may soon achieve cinematic quality.

We’re excited by the possibilities that AI video production will bring and we’re keeping abreast of the latest developments so we can deliver the time and cost-saving efficiencies that AI affords to our clients.

But, as a video production agency, we recognise the threat AI poses to many in our industry and other creative fields. It’s intimidating to say the least, but, to remain relevant in the industry we must adapt and grow. With so many AI tools and use cases, it’s essential to understand where AI can help and where it falls short. Here, we explore the strengths and weaknesses of AI in video production.

Key Takeaways From This Article

• AI’s Role in Video Production:
  • AI aids in idea generation, script overviews, and scene blocking.
  • Useful tool for pre-visualisation, helping directors experiment with visual styles.
• Stock Footage & Editing:
  • AI-generated drone shots and stock footage save time and resources.
  • Effective in object removal, motion tracking, background removal, and rotoscoping.
  • Automates repetitive tasks in video editing, enhancing efficiency.
• Limitations of AI:
  • Struggles with creative script refinement and emotional storytelling.
  • Challenges in generating complete videos autonomously.
  • Inconsistent rendering of hands, fingers, and maintaining overall visual consistency.
• Suitable Tasks for AI:
  • Visual film pitches, initial concept trailers, and pre visualisation
  • Rough storyboarding, scene testing, and visual experimentation
  • Music videos, concert visuals, sci-fi stories, parody, and advertisements
• Ethical and Environmental Considerations:
  • Potential job displacement and erosion of creative professions.
  • Significant computational resources impacting the environment.
  • Copyright issues due to lack of human authorship in AI-generated works.

Where AI Excels

To begin with, let’s have a look at the areas where AI can be a useful tool, which video producers can harness for:

Ideation and Planning

AI has become a valuable tool for idea generation in video production. By providing creative suggestions and conjuring up concepts instantaneously, it can serve as an excellent starting point for any project. Oftentimes, getting off the starting blocks with an idea can be challenging, so having auto generated ideas to springboard off can be helpful in finding that initial spark. In fact at the moment we are using Midjourney to help us visualise ideas for a pitch. 

Similarly, AI is improving when it comes to  writing rough scripts. It can draft script frameworks and early iterations, offering a foundation that human writers can refine and develop further. When it comes to scene blocking, AI proves useful by helping to plan the arrangement of scenes, ensuring that the spatial relationships between characters and objects are coherent. It can also assist in reviewing for continuity errors. By analysing the narrative and visual elements, it can spot inconsistencies that might go unnoticed.

Additionally, when planning pre-visualisation of movie ideas and scenes, AI can simulate how scenes could look, allowing directors to experiment with different visual styles and compositions, before coming to a conclusion on which path they would like to take. This is fantastic for pre-production because it ensures that everyone is aligned on the vision before production begins. 

Stock footage & Editing

AI-generated simulated drone shots will soon be able to provide aerial perspectives without the need for actual drones, or going to foreign locations for the shot, saving time and resources. Additionally, when AI video develops to a more realistic stage, it will be great at producing stock footage. AI also offers tools for stylising video, applying various effects to enhance the visual appeal. AI is already helping to speed up post-production workflows with new features in Premiere Pro.

For post-production tasks, AI is effective in object and background removal in both images and videos, eliminating unwanted elements seamlessly. This is likely because motion tracking is another area where AI excels, allowing it to follow movements precisely, which is extremely useful for special effects and compositing. In rotoscoping, AI can help save time with the frame-by-frame animation process, allowing editors to focus on creative decisions. 

Essentially, AI is most useful to us as video makers in the early phases of video production, aiding in ideation and planning. It can be  extremely useful in post production for repetitive, laborious tasks, leaving room for more creativity from the humans editing.

Where AI Falls Flat

Refinement and Understanding

Despite its strengths, AI still struggles in several areas of video production. One major challenge is creative script refinement. While AI can draft rough scripts, it lacks the nuanced understanding and emotional intelligence required to refine these scripts into compelling narratives. It falls short in showing empathy, an essential element in storytelling that involves truly understanding and conveying human emotions, making AI-generated content difficult to connect with. As well as the limitations for emotional connection, AI also struggles to create lifelike movements and moving human expressions. Achieving realistic human motion and expressions requires a level of sophistication that AI has not yet attained.

Video Generation

Generating video from scratch is an area where AI currently lags. While it can assist in various stages of production, creating a complete video autonomously remains beyond its capabilities. Rendering smooth movements, particularly faces and hands, accurately is a notorious challenge for AI, often resulting in unnatural and distorted images. Additionally, maintaining consistency is another hurdle for AI. Ensuring that elements such as lighting, colour grading, and visual style remain consistent throughout a production can be challenging. While AI is a powerful tool, it still requires a lot of human oversight and intervention to achieve good results.

Projects that AI Could Be Useful For Given The Current Tech

Given its current capabilities, AI is particularly well-suited for projects like visual film pitches, where it can help depict concepts quickly. It could be beneficial for initial concepts and mapping out video, helping with planning shots and pitching ideas as a visual aid. Concept trailers, scene testing and pre-visualisation benefit from AI’s ability to simulate and experiment with different visual elements.

Storyboarding is another good use case, with AI generating detailed visual plans for scenes.  AI also excels in creating music videos, experimental projects, and advertisements, which lend themselves to stylisation and creative effects. Concert visuals and sci-fi/non-organic stories which involve fantastical elements can lean on AI capabilities to be brought to life.

Ethical and Environmental Considerations

While AI offers numerous benefits, it also raises ethical and environmental concerns (which we will touch on in a future article). The use of AI in video production must be carefully managed to ensure that it does not lead to job displacement or the erosion of creative professions. Additionally, the environmental impact of running large-scale AI models, which require significant computational resources, must be considered.

Another consideration is the validity of copyright over AI-generated images. The US Copyright Office recently rejected a request to let an AI copyright a work of art. The board found that the AI-created image didn’t include an element of “human authorship,” (which is a necessary standard for protection). The board’s decision calls “the nexus between the human mind and creative expression” a vital element of copyright. It is notable that copyright law doesn’t directly outline rules for non-humans, but courts have taken a dim view of claims that animals or divine beings can take advantage of copyright protections. 

This doesn’t necessarily mean any art with an AI component is ineligible. If someone tried to copyright a similar work by arguing it was a product of their creativity done by a machine, the outcome might look different. Either way, this highlights the importance of human input and agency in machine-produced works. As AI becomes an ever-growing part of our lives and repertoires, the limits of its copyright could be tested many times in the years to come.

AI is transforming the way we create and edit videos, offering powerful tools that can streamline processes and boost creativity. It provides endless inspiration and offers exciting potential to help with pre-production and planning phases, assisting us with  idea generation, and offers significant advantages in post-production by automating repetitive tasks. However, AI has vast limitations, such as struggles with creative script refinement, emotional storytelling, and generating complete videos autonomously. 

Human input is crucial in video generation, not only for copyright and ethical considerations, but also because the most impactful media today is authentic and understands human wants and needs—something AI is unlikely to ever  fully achieve.

AI could be a powerful ally in video production, but the human touch remains irreplaceable for creating truly engaging and authentic content.

Author Bio

Adam Neale has been working in the video industry for over two decades. He has run multiple production companies and has received awards for his work from global film festivals along with a Vimeo Staff Pick and a Webby Award.