Projects

SSI has and is being used in projects funded by the European Union (EU), the German Research Foundation (DFG) and the Federal Ministry of Education and Research (BMBF):

Virtual Assistant to Support People with Mild Cognitive Impairment (MCI) on the Basis of Smart Glasses Abstract: Mild Cognitive Impairment (MCI) is characterized by limitations in performance related to memory, attention and cognition. Basic daily life skills of people are usually not affected by mild cognitive disorders. However, difficulties may arise when such people have to deal with more complex tasks within their daily life. The project GLASSISTANT makes use of Smart Glasses (such as Google Glass or Epson Moverio) in order to support people with early-stage dementia. The current stress level of the person will be monitored by Wearable Sensors. Based on the interpretation of sensor data (such as pulse), the system recognizes when a person needs most likely assistance. In this case, GLASSISTANT analyzes the field of view of the elderly person and displays information about objects and situations using Augmented Reality Techniques.

Within the project SSI is used for emotion recognition from audiovisual, as well as, physiological and inertial data. To run the detection system on the Google Glass, an android port called SSJ is being developed.

Project page.

The SenseEmotion project aims to optimize pain treatment by developing a multisensor-based system that can automatically detect physical and emotional pain as part of an adaptive affect management system that can reduce pain related emotions in order to improve the quality of life and wellbeing of elderly people.

Within the project SSI is used for synchronous recording in experiments and real-time detection of user’s emotional and physical pain through biopotentials, voice, body and facial expression recognition. The derived user state will then be used by an avatar-based affect management system, that can commence context-sensitive support in the form of crisis intervention and calming measures.

Project page.

The ARIA-VALUSPA project will create a ground-breaking new framework that will allow easy creation of Artificial Retrieval of Information Assistants (ARIAs) that are capable of holding multi-modal social interactions in challenging and unexpected situations. The system can generate search queries and return the information requested by interacting with humans through virtual characters. These virtual humans will be able to sustain an interaction with a user for some time, and react appropriately to the user’s verbal and non-verbal behavior when presenting the requested information and refining search results. Using audio and video signals as input, both verbal and non-verbal components of human communication are captured. Together with a rich and realistic emotive personality model, a sophisticated dialogue management system decides how to respond to a user’s input, be it a spoken sentence, a head nod, or a smile. The ARIA uses special speech synthesizers to create emotionally colored speech and a fully expressive 3D face to create the chosen response. Backchannelling, indicating that the ARIA understood what the user meant, or returning a smile are but a few of the many ways in which it can employ emotionally colored social signals to improve communication.

Within the project SSI is used to integrate state-of-the art facial expression recognition and emotional speech analysis to drive the socially behavior of the character. The non-verbal cues detected with SSI are combined with the transcription provided by a speech recognition engine and shared with the dialogue management system. In the course of the project new techniques such as Deep Neural Networks will be tested and compared to traditional recognition methods. To collect a sufficient large training corpus rich in natural interaction, a multi-user recording system has been developed with SSI, which allows users to monitor users while remotely communicating in physically separated rooms. User interaction is recorded with HD video/depth cameras and room/close-talk microphones for later analysis.

Project page.

KRISTINA’s overall goal is to research and develop technologies for a human-like socially competent and communicative agent that serves for migrants with language and cultural barriers in the host country as a trusted information provision party and mediator in questions related to basic care and healthcare. In particular, KRISTINA aims to develop a companion for patients (or, more generally, individuals with questions about basic healthcare and the healthcare infrastructure in the host country), elderly in need of care, and care personnel who experience problems of interaction due to their migration background.

In KRISTINA, SSI is used to analyse the facial expressions, gestures and voice of a user with respect to displayed emotions. The results of those analysis are then fused within the Framework to produce one continuous output-stream which describes the emotional state of the user at any time. Furthermore SSI is applied in the development of a central handling location which connects multiple cloud services, such as speech-to-text, language-analysis or video-streaming. Those services and the emotion-analysis are providing the necessary input information for the dialog-management in KRISTINA.

Project page.

Job interviews often times pose a great challenge to job seekers. The task of appearing interested, motivated and authentic is difficult especially for young persons. Virtual training system can be used in this context to generate new opportunities for such persons to acquire job interview pertinent social skills. EmpaT aims to combine real-time analysis of social signals with an emotional online user model in order to enable a virtual avatar to react and adapt to the user and the situation during the interaction. The challenge here lies in the generation of realistic feedback for the virtual avatar. EmpaT is funded by the German Ministry for Education and Research (BMBF) and is administrated by VDIVDE-IT GmbH.

Within the project, SSI is used for the analysis of the user’s non-verbal social cues of the interviewee, including motion capture data, facial features and prosody. The virtual agent adapts to the user’s nonverbal behaviour and he or she is given feedback about the performance right after the interview, based on the recognized behaviours within SSI.

Project page.

Logue aims to augment social interactions by providing the user with feedback on her or his own behaviour in realtime using different modalities: visual, auditory and haptic. The goal is to increase awareness and improve the quality of one’s own nonverbal behaviour.
The behaviour of the user is extracted from various sensors and then analysed in realtime using the android port of the SSI framework (SSJ).

Project page.

The CARE project addresses the development of technology for senior users with the aim to improve their general wellbeing. The prototype system named CARE is used for in-situ testing in a senior’s home and combines functionality of a digital image frame with an active recommender mode. Recommendations are chosen on the basis of sensor data and a well-being model to carefully decide on at which point in time what kind of activity will be most suitable to suggest.

In CARE multiple data providers are used to gather sensor data from different sources. SSI acts as one of these sources and is in fact the most important one as plugins for required sensors already existed or could easily be integrated during the project. SSI is accessing, filtering and interpreting multiple sensors in real time that are further analyzed by a higher level interpreter in combination with data from other providers to gain information about the users’ living environment, activity and mood. The data is sent to the CARE system via recent internet of things (IoT) / web protocols that were integrated into SSI as part of the project.

Project page.

TARDIS aims to build a scenario-based serious-game simulation platform for young people at risk of exclusion, aged 18-25, to explore, practice and improve their social skills. TARDIS will facilitate the interaction through virtual agents (VAs) acting as recruiters in job interviews scenarios. The VAs are designed to deliver realistic socio-emotional interactions and are credible, yet tireless interlocutors. TARDIS exploits the unique affordances of digital technology, by creating an environment in which the quality and the quantity of emotional display by the agents can be modulated to scaffold the young trainees through a diverse range of possible interview situations. The scenarios are co-designed with experienced practitioners in several European countries in order to ensure their relevance to the different individuals across a number of cultural contexts.

Within the project SSI is used for real-time detection of user’s emotions and social attitudes through voice, body and facial expression recognition. The derived user state is then used to adapt the progress of the game and the virtual interlocutor’s behaviour to the individual users. Thus facilitating reflection on their own practice and enabling a more flexible and personalised coaching for young people at risk of social exclusion.

Project page.

Laughter is a significant feature of human communication. ILHAIRE objectives are to help the scientific and industrial community to bridge the gap between knowledge on human laughter and its use by avatars, thus enabling sociable conversational agents to be designed, using natural-looking and natural-sounding laughter. will gather data on laughter using high quality sound, video, facial and upper body motion capture. The process of database collection will be grounded in psychological foundations and the data will be used to validate computational and theoretical models of analysis and synthesis of audio-visual laughter. Dialog systems between humans and avatars will be developed and we will conduct studies to capture the qualitative experience evoked by a laughing avatar.

Within the project SSI is used for large data collections involving several users and different type of sensors (audio, video, motion capturing), as well as, for real-time detection and classification of laughter by combining the various information sources.

Project page.

The Collective Experience of Empathic Data Systems (CEEDs) project aims to develop novel, integrated technologies to support human experience, analysis and understanding of very large datasets. In a wide range of specialist areas – such as astronomy, neuroscience, archaeology, history and economics – experts need to make sense of and find meaning in very large and complex data sets. Finding meaningful patterns in these large data sets is challenging. Therefore, CEEDs will develop and use a wide range of unobtrusive multi-modal wearable technologies to measure peope’s reactions to visualisations of large data sets in specially built virtual, or synthetic, reality environments. By monitoring these measures, CEEDs will identify users’ implicit (subconscious) responses to different features of visualisations of massive datasets. The implicit responses will then be used to guide users’ discovery of patterns and meaning within the datasets.

Within the project SSI provides the sensing platform where user signals (heart rate, skin conductance, eye gaze, …) are captured and pre-processed. In a second step explicit and implicit user behaviour is extracted and collected by the sentient agent, which variates the presentation to the users needs – creating a loop between the system, presented data and the user.

Project page.

The DFG-funded project OC-TRUST aims to investigate trustworthiness of Organic Computing (OC) systems. Within this project, different partner work on models, metrics and algorithms to establish trust between device components of OC systems (device trust) and trust of a user in OC systems and to other users (user trust). Of particular interest is to investigate trust in the context of human-computer interactions (HCI) with OC systems. For instance, trust models and metrics are researched to determine and predict user trust in OC systems. Additionally, a repertoire of approved evaluation methods and techniques is established in order to conduct user studies and determine user trust.

Within the project, SSI is used for the continuously measurement of trust-related user behavior (attention, engagement) by means of physiological data. These data include the user’s eye gaze fixation time and heart rate. By analyzing these signals our methods provide important knowledge about the users and their behavior during the interaction. This knowledge is then used to manage user trust automatically by means of self-adaptations, such as by presenting more system transparency and user control.

Project page.

CALLAS aimed at closing the gap between the emerging capacity of conveying emotional aspects within multi-modal interaction and the growing expectations of people for a more natural and pervasive interaction with digital media applications in intelligent adaptive spaces. A framework was developed based on a plug-in multimodal architecture, invariant to configuration of Multimodal Components, to interpret and process emotional aspects in real-time for easy and fast development of applications for Art and Entertainment, paying attention to the value of users, who are no longer passive spectators of artistic performances, but stimulating sources of human communication.

Many ideas that have driven the design of SSI have come from the needs arised in the CALLAS project, e.g. accurate synchronization of modalities, fast and efficient filtering of raw input signals, possibility to retrain models to a specific enviroment.

Project page.

At the Music Hack Day 2013 in Barcelona we used SSI to develop an old school jump and run game called ‘BlowUp’. ‘BlowUp’ is steered with only physiological sensors and uses the eHealth board for Arduino to capture air flow and heart beat of the user. Signals are pre-processed and streamed through SSI and sent to the game. During the game the character has to avoid obstacles by jumping controlled through air flow, while its speed is adjusted to the heart of the user. If the heart rate increases the speed of the character goes up and the other way round.

Project page.