Michelangelo Logo
MICHELANGELO
New technology to help children with Autism

MICHELANGELO Project at the ITASD 2014 Conference

MICHELANGELO at the ITASD

The 2nd International Conference on Innovative Technologies (IT) for Autism (ASD) was held in Paris at the Pasteur Institute on October 3-4, 2014. The ITASD conference was organized by the Orange Foundation, the Adapta Foundation, Autism Speaks and the Pasteur Institute.

This was the second appointment after the successful 1st edition of Valencia in Spain (2012).

The theme of the Paris event was very direct and clear: “Digital solutions for people with Autism” i.e. “how digital technologies can be used to improve the daily lives of people with autism”.

A key characteristic of ITASD is the capability of bringing together families and professionals and allowing them to share needs and experiences, exchange best practices and discover innovations.

During the two days program the Conference included plenary sessions, parallel sessions and a panel discussion, poster presentations and demos. A comprehensive range of topics were addressed and discussed evidencing the added value of technology for the Autism in:

  • Early diagnosis and assessment,
  • Remote management, treatment and rehabilitation,
  • Education and learning,
  • Employment and inclusion into the labour market,
  • Communication skills,
  • Socialization.

Advances in specific technologies /applications such as serious gaming, mobile apps for tablet PCs and smartphones, behaviours and emotions recognition, sensors for the monitoring of activity and physiological signs, robot, eye tracking were presented.

The MICHELANGELO project was very well represented at the conference with five papers and two posters.

Here below we report their abstracts.

 

Session: EARLY DIAGNOSIS AND RESEARCH INTO THE NATURE OF ASD

Authors: Salvatore Anzalone , Boucenna B, Tilmont E, Xavier J, Jouen AL, Chetouani M, Cohen D.

Title: Exploring imitation and joint attention of children with autism spectrum disorder during interaction with a robot

Background: 

Joint attention (JA) and imitation are two important skills for early development of social interaction and are both targeted in treatment of children with autism spectrum disorder (ASD).

Objectives: We aimed to study JA and imitation using information communication technologies and to develop specific metrics related to quality of interaction.

Methods: 

We performed two experiments of interaction with a robot involving 16 children with ASD (mean age=9.25 (±1.87) years, 12 males and 4 females) and 16 matched children with typical development (TD).

Results: The JA experiment showed that both groups of children performed well with the therapist. However, with Nao, both groups had lower JA scores, and the children with ASD had a significantly lower score than the TD children showing that JA skill depends on the interaction partner. We also found that (i) multimodal JA induction (gazing + pointing + vocalizing) was more efficient in both groups; (ii) the 3D spatial world gaze exploration showed less accuracy and the trunk position showed less stability in ASD. In the second experiment, a robot learned different postures by imitating several partners (adults, TD children and children with ASD), during an imitation game. Robot learning was based on a sensory-motor architecture whereby neural networks enabled the robot to associate what it did with what it saw. Several metrics (i.e., annotation, the number of neurons needed to learn, and normalized mutual information) were used to show that the partners affected robot learning: (i) learning was easier with adults than with both groups of children, indicating a developmental effect; (ii) learning was more complex with children with ASD compared to both adults and TD children.

Conclusion: 

Contrarily to expectation, spontaneous understanding of JA with the robot was not immediate. However, imitation game with the robot was easier and similar to human partner interaction. Given the discriminative performance of the number of neurons needed to learn, we propose it as a quantitative tool to study  imitation in clinical trial.


 

Authors: Lucia Billeci; Crifaci G.;Narzisi A.; Campatelli G.; Pioggia G.; Muratori F.

Title: Eye-tracking technology to assess joint attention deficit in children with Autism Spectrum Disorders

Background:  

Autism Spectrum Disorders (ASD) are a set of development disorders characterized by a significant disturbance of social development. Joint attention (JA) abilities play a crucial role in the development of ASD. Impairments in JA are among the earliest signs of the disorder and JA skills relate to outcome, both in the ‘natural course’ of ASD and through being targeted in early intervention programmes. JA skills typically emerge within the first year of life in normally developing children. Therefore, deficits in these skills may mark a fundamental aspect of the developmental disturbance that characterizes ASD. Eye-tracking assessments of JA may provide more precise spatial and temporal information than face-to-face assessments.

Objectives:  

Application of eye-tracking technology to evaluate response and initiative to JA in children with ASD compared with typically developing peers (No-ASD). 

Methods:  

The methodology was tested on 11 ASD and 13 typically developing children (age 2-6 years). The gaze scan path of the children were recorded by a SensoMotoric Instruments IView X RED table-mounted 120Hz eye-tracker, integrated with a 22’’ monitor, while children were seated in front of the monitor. Experiments began with a 5-point eye-tracking calibration procedure with targets consisting of small figure presented together with contingent sound. This was followed by the presentation of children's video sequences (9 s). The experiment consisted of three tasks: a response to JA (RJA) and two initiative to JA (IJA) tasks. The RJA task consisted of a woman between two identical objects placed in front and on either side of her; she smiled, said ‘‘Hello’’ and turned her head toward one of the two objects. In the two IJA tasks the woman, after said “Hello”, maintains direct gaze but in one case one of the object actives unexpectedly, while in the other one the object appears from one end of the frame and crosses the scene. Different Areas Of Interest (AOIs) were defined: person’s face, eyes, mouth and body, target object and non-target object. For each AOI the fixation count (FC), the first fixation duration (FFD) and the fixation time (FT) were computed.

Results:  

 In the task 1 FC and FT were increased for the eyes and the FFD for the target object in the control group with respect to the other AOIs. In the task 2 FFD increased for the mouth in the control group with respect the ASD, instead in the task 3 FDD increased for the target object in ASD with respect to the control group. Comparing the different tasks we observed an increased FC and FT in task 2 vs task 1 and 3 for target object, anincreased FT in task 1 vs task 2 and 3 for eyes in controls.

Conclusions:  

The ASD children pay less attention to eyes; instead they seem to be more attractive by the target object especially in RJA. Eye-tracking measurement of JA is a promising prognostic tool because it is highly precise and standardized and it is used to better characterise JA in young children with ASD. 


SESSION: SUPPORTING EVIDENCE-BASED APPROACHES FROM DIFFERENT RESEARCH PERSPECTIVES

Authors: Antonio Narzisi; Tilmont, E; Jouen, A; Cohen, D; Muratori F

Title: Patient-Centric Model for Remote Management, Treatment and Rehabilitation of Children with Autism: Michelangelo Project Study Protocol

Background: 

The interaction between clinical research and technology in autism spectrum disorder (ASD) is essential for the development of rehabilitation tools. The MICHELANGELO project intends to bring the therapy of autism out of the clinical environment and to develop a patient-centric intervention. Objectives: The aims are: (1) to demonstrate the usefulness of Information Communication Technology (ICT) in the rehabilitation of children with autism; (2) to appreciate the contribution of ICT tools, such as a digital tablet for serious game, in terms of easy at-home use. 

Methods: 

For the purpose of this study, 20 HFASD children will be recruited. They will be divided into two groups: an Experimental group (N=10) and a Control group (N=10). The Control group will not go through the training sessions, they will participate only in the assessment sessions. Training session will be organized in two periods: (a) one-to-one clinical session (based on imitation and joint attention) at hospital; (b) serious game session both at hospital and at home. The games, based on imitation and joint attention tasks, were designed with developmental foresight to offer various media appropriate to the level and abilities of each child.

Results:

An observation room and a semi-naturalistic protocol based on imitation and joint attention tasks, during which EEG and ECG data were developed. To follow a naturalistic philosophy for EEG recording ENOBIO system has been used. This system offers a high degree of unobtrusiveness. Its active digital electrodes digitize the signal on site in order to reduce environmental noise and wireless communication while recording data away from the lab or controlled environments. Children wear also the chest strap called SHIMMER. It registers the heart's electrical signal. The system is able to visualize the normal sinus rhythm, arrhythmias and to assess heart rate (HR) and heart rate variability (HRV). Moreover Child and Therapist will wear two Head Mounted Micro Camera (called eye-scene) in order to detect the ‘trigger event’ (i.e. eye contact between the child and the therapist). 

Conclusions: 

The aim of the treatment program introduced here, is to overcome some limitation of the present applications of treatment neurofeedback-based in particular the artificial and constrained situations in which data are acquired and the need of an effort from the subjects that is difficult to obtain treating very young autistic children. It has been observed that young children accepted the EEG cap and the protocol and they were able to perform the tasks without difficulties or constraints. The interaction with the examiner rather than with a screen is very important for recreating a more real situation with social interactions and cues. In this situation we can suppose that the signals acquired from the brain, are much more similar to that generated while the child interact in common life situations. 


Authors: Federico Cruciani; D’Amico, D; Donnelly, MP; Galway, L; Paggetti, C, Nugent, CD, Tamburini, E..

Title: The Dante System: A Video Annotation System to Support the Therapist in Behavioural Assessment

Background: 

Evolving technologies and wireless wearable sensors can provide rich contextual information to support treatment for ASD. In this paper, we present a system supporting behavioural characterization through video analysis, giving the therapist contextual information regarding the physiological state of the child gathered through wearable sensors. The Dante system is tailored for the MICHELANGELO Project, whereby EEG and ECG wearable sensors are coupled with video during therapy sessions. However, the entire system is adaptable, permitting the use of any physiological sensor or additional data source.

Objectives: 

This work aims to provide a behavioural assessment tool for therapists, simultaneously providing simplified data access for the analysis of possible correlations with recorded physiological signals. The main objective of Dante is to facilitate behavioural assessment, making the annotation process partially automated through the automatic detection of certain features such as Eye Contact and Joint Attention. Furthermore, quick access to contextual physiological data for further analysis is supported by the platform.

Methods: 

During a therapy session, audio and video are recorded using a microphone and two environmental cameras, focused on the child and on the therapist. In addition, two small wearable wireless cameras are used to support automatic detection of certain behaviours. Automatic detection comprehends object and colour tracking to analyze the child’s movements during the therapy session. Additionally, the wearable wireless cameras permit detection of Joint Attention Request and Response, providing an estimation of the Eye Contact between therapist and child through face detection on the video signals.  Video signals are also shown in real-time on a screen in a separate room, which permits researchers to flag portions of the session that exhibit behaviours that could not be detected automatically. Post session, researchers can playback the recorded video and provide manual annotations for this broader range of behaviours. This characterization is based on a coding scheme focused on the main evaluation objectives of the therapy: Imitation and Joint Attention (Table 1). The system provides synchronized storage of Audio/Video, EEG and ECG signals, allowing fragments of EEG or ECG data to be exported. In this way, it is possible to isolate data portions for further analysis.

Results: 

An exploratory study using the MICHELANGELO platform was conducted at APHP, in Paris, France and at IRCCS Stella Maris, in Pisa, Italy. The subject of the study were 20 children diagnosed with ASD. Researchers were able to characterize the behaviours that emerged during each session, using the automatic and manual annotation features. The system meets the objectives and succesfully supports a streamlined procedure for the recording and production of annotated sessions, which comprise data from heterogeneous sources, such as audio, video and EEG/ECG sensors. 

Conclusions: 

Even though the system will be extended with further functionalities, both adding new video analysis functions, as well as integrating other sensors; the platform’s current use within the MICHELANGELO project appears to be more than a proof of concept. However, we expect further input for development arising from the study undertaken in Pisa and Paris.


 

SESSION: SERIOUS GAMING AND VIRTUAL ENVIRONMENTS

AuthorsValentina Bono ; Jouen AL ; Narzisi A ; Jamal W ; Tilmont E;  Maharatna K ; Cohen D. ; Muratori F. 

Title: A Novel Serious Gaming Platform for Patient-centric Characterization of ASD children 

Background: 

Patient-centric characterization and home-based intensive intervention in accordance have been considered key elements in treating children with Autism Spectrum Disorder (ASD). 

Objectives: 

We aimed to develop a novel serious gaming platform that enables cognitive characterization in patient-centric way during execution of imitation and Joint Attention (JA) tasks and deliver tailored therapy at home settings for fostering intensive intervention. 

Methods: 

We developed a multiplayer serious gaming platform based on the Early Start Denver Model (ESDM) stimuli involving a child, called Player, and a therapist/parent/carer acting as a Master. It uses two touch-screen tablets/computers – one for the Player and one for the Master – communicating in real-time facilitating interactive online gaming. A novel video-based technique adopted here emulates human-human interactions. Each game consists of different difficulty levels. The nature of the games and the stimuli they map are shown in Table 1. 

Results: 

Qualitative evaluation at two sites with 10 ASD children showed high-level of engagement/enjoyment even for the activities they refused to execute previously in non-computerized environment. Embedded time-stamping, automated correctness checking and event correlation procedures enable measuring a 

set of quantitative metrics in task- and difficulty-specific way for facilitating complete characterization of a child based on which patient-specific treatment could be planned.  

Conclusion: 

The proposed platform offers an effective way for characterizing ASD children according to their ability in imitation and JA tasks. Owing to the flexibility of the platform other forms of stimuli could be easily integrated allowing a wider range of characterization objectives. The platform could also be directly used for implementing intensive intervention at home.


POSTERS

Authors: Donnelly, MP; Galway, L; Cruciani, F; Raptis, S; Nugent, CD, McCullagh, PJ, Lightbody, G, Paggetti, C and Bonfiglio, S.

Title: The MICHELANGELO Platform: A Simple Framework to Support Engagement with Complex Technologies

Background: 

MICHELANGELO is a EU funded project that aims to utilize a range of emerging healthcare technologies (video, activity, heart and brain-wave monitoring, robotic and tablet-based serious game environments) to augment both clinic and in-home based intervention strategies for children with autism spectrum disorders. This paper presents the underlying communication architecture used to connect and manage these different technologies.

Objectives: 

The paper introduces the MICHELANGELO Platform prior to detailing the main touchscreen-based interfaces that enable parents and clinical staff to easily control the underlying complex technologies within the system. A key aspect of the MICHELANGELO Project is the investigation and analysis of potential correlations between exhibited behavioral characteristics and neurological activity, observed during therapeutic interventions. As such, within the paper we also present the main web-based clinical decision support interface that intuitively summarizes and presents behavioral and physiological trends over time, to enrich clinical staff understanding of progress.

Methods: 

The MICHELANGELO Platform provides a common infrastructure, based on XML and web-based protocols, to simplify the underlying communication with the range of Bluetooth, Wi-Fi, wired and software-based devices that provide video, activity, electrocardiography (ECG) and electroencephalography (EEG) monitoring, as well as robotic and serious-game based interventions in home and clinic environments (Figure 1). The infrastructure is flexible enough to support this range of heterogeneous data sources, and facilitates the future inclusion of new sensor / device types. Furthermore, it also supports the personalized configuration of device components for each end-user. Sitting on top of this communication layer is a suite of graphical user interfaces that significantly reduce the technical complexity of operating the system. Users are presented with an intuitive touchscreen interface (Figure 2), deployed on a Windows tablet, that enable them to manage therapy sessions including, support for remotely starting and stopping the different devices, launching the serious games platform and event logging (recording events that represent potentially important behavioral responses displayed during a session). Consequently, a web-based clinical decision support interface (Figure 3) supports the exploration and analysis of synchronized behavioral and physiological parameters. All sensor / device data and associated events can be summarized to provide a periodic review of a child’s progress. Based on a clinician’s assessment of progress, this clinical decision support interface also facilitates the modification of a child’s personal intervention protocol.

Results: 

The MICHELANGELO Project is currently managing user evaluations in both Italy and France, where 20 children (10 controls) have been recruited to participate. The participants will be engaged in an early intervention approach following the Early Start Denver Model that will be augmented by the aforementioned technology platform. Preliminary results emerging from engagement with the MICHELANGELO Platform will be reported in the paper.

Conclusions: 

The MICHELANGELO Project aims to support increased understanding of autism through the provision of ICT tools for interrogating the correlations between behavioral characteristics of child and time-associated physiological responses. In this paper, we present the flexible platform that supports personalization and in particular, present the suite of graphical user interfaces that help to significantly reduce the technical complexity of using the system.


Authors: Tonacci A; Corda D; Billeci L; Narzisi A; Muratori F; Pioggia G.

Title: An unobtrusive platform for the monitoring of the autistic child at home

Background: 

The design and realization of a platform to monitor physiological parameters of human body represents a challenge in studies about Autism Spectrum Disorders (ASD). In particular, Electrocardiogram (ECG) is among the most useful signals that can be extracted by such a similar system.

ECG is employed to assess the heart rate and its variability, often associated with several important pathologies (heart abnormalities), and to particular psychophysical states (stress and relax).

The ECG monitoring is important to classify the arousal status of ASD patients, in order to plan and adopt customized, patient-centric therapeutic approaches to sensitively increase the treatment efficiency.

Objectives: 

To design a real-time physiological monitoring system and to validate it during a Joint Attention task in ASD and typical developing (TD) controls. 

Methods: 

The Real Time processing of the ECG RT (Recording Tool) included an ECG storage for a further visualization and analysis, a features extraction (extracting the most relevant features chosen a priori), and storage (into an .xml file), and a supervision of possible events occurring within the acquisition phase. Such events are sent in real-time to an RTM (Recording Tool Manager), together with a status message to reply to the status check eventually sent by the RTM to the RT. The overall architecture of the real time system is shown in Figure 1. 

We assessed the ECG signal in 11 ASD children and 13 age -matched controls. Statistical analysis was performed on time and frequency domain parameters by using SPSS Version 20 software.

Results: During baseline, a significant difference (p=.02) was found between ASD and TD in LF/HF ratio (LF: low frequency, HF: high frequency) , suggesting an increased sympathetic activity in ASD. On the other hand, TD showed an increment (p=.04) of respiratory frequency, calculated as the peak of HF components of the ECG signal.

During the task, LF/HF ratio showed a trend very similar to the baseline, with higher values for ASD (p=.06).

The mean HR (Heart Rate) decreased both for ASD and TD during task, with higher value recorded in ASD children.

As for Heart Rate Variability (HRV) features, they revealed a decrease during the task in both the population, with a slightly more marked slope in the case of ASD. The LF/HF ratio showed an evident increase during the task for ASD, both determined by an increase in the LF and a decrease in the HF, while TD did not evidence a similar trend, but only a slight increase in this feature.

Interestingly, the Respiratory Sinus Arrhythmia (RSA) analysis demonstrated that TD showed the typical RSA trend, with a significant decrease during the task with respect to the baseline, while ASD exhibited the opposite trend, with a similar increase during the task, as largely expected. 

Conclusions: 

The results obtained demonstrated the feasibility of a minimally obtrusive, portable platform for real-time monitoring physiological signals, such as ECG, in children affected by ASD, representing an useful tool for physiological monitoring also in ASD and related syndromes. 

Presentation Videos

Salvatore M. Anzalone
University Pierre & Marie Curie, France  
 
Exploring imitation and joint attention of children with autism spectrum disorder during interactions with a robot
 
Click to view
Lucia Billeci
Institute of Clinical Physiology National research Council of Italy (CNR) , Italy  
 
Eye-tracking technology to assess joint attention deficit in children with Autism Spectrum Disorders
 
Click to view
Antonio Narzisi
IRCCS Stella Maris Foundation, Italy  
 
Patient-Centric Model for Remote Management, Treatment and Rehabilitation of Children with Autism: Michelangelo Project Study Protocol
 
Click to view
Federico Cruciani
I+ S.r.l., Italy

  
 
The Dante System: A Video Annotation System to Support the Therapist in Behavioral Assessment
 
Click to view
Valentina Bono
University of Southampton, UK  
A Novel Serious Gaming Platform for Patient-centric Characterization of ASD children
 
Click to view
top top
Copyright © 2012 - 2014 Michelangelo Project - Helping Children with Autism
The Michelangelo Project is co-funded from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° #288241