Making comic books from movies using computer vision

The automated system, inspired by Japan’s ‘Manga’ comics, will help professionals develop their own books from videos, movies, TV shows or cartoons

The team conducted several experiments to compare results to that of existing state-of-the-art techniques.

(Subscribe to our Today's Cache newsletter for a quick snapshot of top 5 tech stories. Click here to subscribe for free.)

Producing a comic book is a work of art involving high level of creativity. The whole process of ideating, sketching, colouring, inking and lettering content, can take over eight months. Now that production can be completed in lesser time, and done by artificial intelligence (AI), not humans.

A team of researchers, including those from China’s Dalian University of Technology, has developed an AI system to make comic books from movies and other videos. They have used computer vision, a form of AI, to automate creative tasks, which helps to interpret digital content like images, videos, and graphics.

The team said it derived inspiration from ‘Manga’, a style of comics or graphic novels originating from Japan. The automated system will help professionals develop their own books from videos, movies, TV shows or cartoons.

The approach will first extract informative keyframes by analysing the subtitles of the video, and then curate comic-style images. A separate multi-page layout framework can allocate the images across multiple pages and synthesise visually appealing layouts based on the relationship between the images, the team noted in a study titled ‘Automatic Comic Generation with Stylistic Multi-page Layouts and Emotion-driven Text Balloon Generation’ .

Then, instead of using the traditional balloon format prevalent in most comic books, the system will generate different types of word balloons based on the emotions of the subtitles, audio and video.

The team conducted several experiments to compare results to that of existing state-of-the-art techniques. It also used different movies and shows like ‘Titanic’, ‘Friends’ and ‘Up In The Air’ to extract clips.