At Captions, we started playing around with the idea of an AI editing your video for you, and it provided a chance to experiment with the notion of collaborative Human-Agent Interaction. We explored how AI can assist in editing your video while still allowing humans to maintain control and direction.

Video editors are software tools in their truest form, with editing primitives such as trimming, adding keyframes, animation curves, etc. We learn what these tools can do for us and how to use them, and finally learn when to apply certain primitives based on our own creative direction and the outcome we’re hoping to achieve.

When you’re letting an AI edit your video for you, you might agree with all, a few, or none of the changes an AI applied. What’s important, is that when an AI is editing a video for you, the AI is using the same human video editing primitives we already know and have a GUI for. Whether that’s how it’s done technically or not, doesn’t matter too much for the end-user experience, but could help with the implementation nonetheless. What matters is how it’s perceived by the end-user, and most importantly, if it allows the end-user to get to their intended outcome faster.

By letting the AI work using the same video editing primitives, we can keep the video editing process collaborative and allow humans to come back and tweak the work the AI did. The AI and the human are speaking the same language by letting an AI use the same primitive tools that humans have used and still can use. This allows us to work faster and allows us to keep our creative control. With this approach, we’re not eliminating any creative human input, we’re eliminating a learning curve, and in fact, widening the addressable audience for these types of creative tools.

For AI Edit, this meant leveraging the set of primitive editing tools we already have in the Captions mobile app and showing where an AI “decided” to apply those primitives. This involved not just showing our users where the actual primitives are being applied as the AI is doing its work (video above), which of course looks cool, but also involved pointing our users to the respective editing primitives in the action bar of the app, making it easy to quickly remove a Zoom, Sound, or Image that was added.

Showing how an AI uses those same primitives allows our users to more easily understand what the AI is doing and, more importantly, how to undo or tweak it. This also helps new users understand how certain video editing primitives affect the final outcome, making the video editing process faster, more collaborative with AI and our over-all product more accessible to new users.

Our work at Captions is hopefully just one step in the direction of AI working collaboratively with creative professionals and consumers to build the best AI-powered products possible. We don’t believe AI will replace “creative professions that never should have existed in the first place“, our intent and approach come down to enabling creatives more so than disrupting them.

TechCrunch launch coverage: https://techcrunch.com/2024/06/26/video-editing-app-captions-releases-a-new-ai-edit-feature-that-automatically-adds-effects-to-your-video/