Creativity-Injection into AI-Powered Multimedia Storyboards

Recently, a large variety of advanced yet simple applications have been trying to democratize complex activities like film-making,
photography and music-making. Specifically, video and photography applications provide editing features similar to those
used in professional software, but easily exploitable by anyone. On the other side, professional software also helps freelancers in their daily activities. However, the key features provided by both creative applications and professional software mainly address the traditional computational aspect with a low level of attention devoted to the creative aspects. It is very easy and common to get a video based on multimedia content stored in our mobile phone, however, the storyboard is mainly based on either temporal and local aspects and not on semantics as an actual human would do. Since creativity (i.e., the use of imagination or original ideas) means different things to different people, an interesting research direction is to model the creativity features of users and translate them into automated procedures to enhance the unsupervised production of storyboards in the context of film-making,
photography, and music-making.

To support the automated production of storyboards in the context of film-making, photography, and music-making, we target
advanced semantic features based on a creative-aware AI approach, whose research objectives are In the following.

Data modeling and classification.
To understand semantic-related aspects of multimedia content, a set of machine learning algorithms will be used by combining
traditional and unconventional approaches to model underlying data content with key components able to model creativity
aspects.

User creativity profiling.
In the literature, several alternative algorithms are available for performing user profiling in different contexts (e.g., recommender systems, cross-selling), and in most cases, no algorithm is universally superior. Furthermore, the profiling of users in terms of creativity is still an open issue, and in many contexts, it is totally disregarded. To address user creativity profiling, a set of artificial intelligence algorithms should be identified and/or properly combined to correctly model the problem. In addition, a set of  interestingness metrics will be studied to evaluate and compare the meaningfulness of the discovered knowledge.

Methodologies to infer creativity-based production storyboards.
Different machine learning algorithms will be studied to derive the key components characterizing creativity-based storyboards
tailored to different applications. Proposed methodologies should exploit semantic-aspects of multimedia content along with user
creativity profiling to provide innovative approaches with respect to the state-of-the-art applications mainly based on temporal and location aspects.

Creativity-based production storyboards with human-in-the-loop.
To enhance the creativity-based production of multimedia storyboards, the proposed machine learning algorithms will be
enriched with self-learning capabilities exploiting user interactions with the presented multimedia “stories” to collect feedbacks and re-train models for the discovery of interesting end-goals. By providing a way of exploiting user interactions, the overall system can be easily customized and adapted to different application scenarios, while the overall design is kept as general-purpose as possible.

During the 1st year, the candidate will study state-of-the-art techniques addressing the enhancement of the creativity patterns
of data and its main algorithms. A set of descriptive metrics will be defined to model data distributions. Since several aspects influence data distributions, the Phd student will draft for each data type (e.g., numerical data, categorical data, short text) an innovative criterion to model data distribution by exploiting unconventional statistical indexes. Furthermore, the candidate will study and define an innovative optimization strategy to properly configure data analytics algorithms.

During the 2nd year the candidate will define innovative metrics to select the most creative patterns that can be effectively
transformed into actions by domain experts. A first release of the creative storyboard will be defined to provide end users with
knowledge presented in a human-friendly way.

During the 3rd year the candidate will study and define a novel algorithm to support a self-learning methodology exploiting a KDB
(Knowledge DataBase) able to select the most relevant and meaningful knowledge according to feedbacks from users, both
active (human-in-the-loop analytics) and passive (data storyboard navigation patterns).

During the 2nd-3rd year the candidate will assess the proposed solutions in real-life application domains based on the candidate
interest, research contracts, funded projects, and available datasets. All these application contexts will be key to disseminate
the proposed solutions not only in the primary venues  (conferences and journals), but also towards industries and the
society, fostering cross-domain fertilization of research results, and new collaborations with other research stakeholders.