IIT Hyderabad builds dataset to understand online user-engagement

DAiSEE documents and annotates four complex markers of engagement

Updated - June 16, 2018 04:33 pm IST

Published - June 16, 2018 03:59 pm IST

  The annotated datas can be used by deep-learning frameworks employed in AI to learn the model accurately.

The annotated datas can be used by deep-learning frameworks employed in AI to learn the model accurately.

Understanding user engagement in online interactions is important in many contexts, with online shopping, advertising, e-learning and healthcare being just a few sectors. Now, IIT Hyderabad has built DAiSEE (Dataset for Affective States in E-Environments), the first multilabel video-classification dataset for recognising boredom, confusion, frustration and engagement. The dataset comprises 9,068 video snippets captured from 112 individuals. For each of these affective states, there are further four levels of labels – very low, low, high and very high. These labels are provided by observing the viewer’s reactions.

Multiple labels

There can be multiple labels assigned to a snippet: “For example, when understanding some complex terminology from videos, a person could display high engagement and still be confused or frustrated at the same time,” explains Vineeth N. Balasubramanian of Department of Computer Science and Engineering at IIT Hyderabad who has led the research. “The combination of data and annotations related to user engagement sets the platform for DAiSEE as a specialized dataset,” he adds in an email to The Hindu. The dataset is available to the public at the website http://www.iith.ac.in/

~daisee-dataset/ Recognising, interpreting, processing and simulating human affective states, or emotions, is an important area of research known as affective computing.

The usual emotions studied include anger, disgust, fear etc. “For a large part, researchers have focused on these basic expressions, we chose to go beyond,” says Dr Balasubramanian.

For instance, in a classroom, the student could be engaged with the lesson, or bored, frustrated or even confused. “Subsequent affective states can be viewed as a result of these four,” says Dr Balasubramanian. For instance, if a person is bored or confused, they could be distracted easily. “The affective states we have considered in DAiSEE are a bit more subtle than the six basic expressions,” he adds.

In the study, people were invited to voluntarily participate in an experiment whether they would watch certain videos and then respond to a questionnaire. The consent of the participants to share the videos was taken. They were shown one educational video and one recreational, so that both focused and relaxed settings may be captured. This gave the researchers 9,068 videos of 10-second length, from which they extracted 27,000,000 images/video frames. “This is larger than most contemporary video datasets,” says Dr Balasubramanian.

Crowd voting

The researchers used a crowd-voting method to annotate the dataset, and the best possible answers were picked using a statistical aggregation method (the Dawid-Skene aggregation). This uses an algorithm to consider the quality of the responses, which are then weighted accordingly to compute the resultant response.

The annotated data can be used by deep-learning frameworks employed in AI to learn the model accurately. In many applications, it is important to learn user engagement so that the algorithm can respond and interact with the user. “We hope for DAiSEE to be a large stride in the direction of promoting a health and improved experience of personalized interaction with such digital systems,” he says

0 / 0
Sign in to unlock member-only benefits!
  • Access 10 free stories every month
  • Save stories to read later
  • Access to comment on every story
  • Sign-up/manage your newsletter subscriptions with a single click
  • Get notified by email for early access to discounts & offers on our products
Sign in


Comments have to be in English, and in full sentences. They cannot be abusive or personal. Please abide by our community guidelines for posting your comments.

We have migrated to a new commenting platform. If you are already a registered user of The Hindu and logged in, you may continue to engage with our articles. If you do not have an account please register and login to post comments. Users can access their older comments by logging into their accounts on Vuukle.