AI-Generated Video: Has AI-Generated Video arrived?

Overview

In our latest round of experiments, we set out with two main objectives. First, we aimed to determine the feasibility of Al generating a 20-second video that merges our unique content with generic data. Second, we sought to gauge the current state of quality of Al-generated videos, specifically assessing its capability to achieve our first goal.

Through these experiments, we gained significant insight into the practical possibilities and limits amid the rapid evolution of Al-generated video.

Key Concepts

Initially, our task involved blending and rendering a custom object, integrating the Red Marble Connectome with a hot air balloon. Using this integration, we transitioned to the video generation phase, utilising the image to produce a video.

This stepwise approach helps in methodically navigating through the process, for precise and enhanced visual representations in the generated video.

Deep Dive

Due to the absence of the Connectome in the base research data, image supply is crucial, enabling a controlled outcome but limiting organic experimentation.

There are a limited number of video generation Al services that utilise images as input to generate videos. Among these, RunwayML emerges as the frontrunner. Other services like Pika and Genmo also offer video generation from image and/or text description inputs, albeit with lower levels of effectiveness and quality.

Setup & Testing

We explored how PlaygroundAl and RunwayML handle custom object blending and rendering. A notable demonstration was done using RunwayML, where a “custom model” was trained to render a flying Connectome hot air balloon, showing a real-world application of custom object rendering.

Additionally, we used RunwayML to creates a video within a time frame of 4 seconds (extendable to 16 seconds). This indicates a certain level of control over the video generation process, making it a structured yet limited approach.

Hitting The Limits

ControlNet struggled with text instructions for object removal or replacement in images. Pika’s video length was unfortunately capped at 3 seconds. Genmo’s video quality was subpar compared to RunwayML and Pika, often significantly altering objects and backgrounds.

Meanwhile, RunwayML had the ability to extend videos by 4 seconds each time, but this caused the source object to change in unexpected ways as the video progressed, morphing undesirably with each extension.

What’s Next? 

Al image generation can create captivating images with rich details, but video generation is still in its early stage. We’re looking forward to exploring more, diving into publications from RunwayML, Meta, and Google. We will also explore open-source tools and other platforms to see what’s on the horizon.

What’s the verdict?

Right now, Al-generated video isn’t yet a controllable medium, but rather an intriguing form of abstract art, captivating but unpredictable.

The tech landscape is teeming with image-focused Al providers, leaving video somewhat in the shadows.

If you’re looking to incorporate your own creative spin or use your own imagery or non-trained data, be prepared for some challenges.