Work package 9 leads the evaluation and impact assessment of the project technology trial. It defines a layered approach to evaluate the technical and organizational structure of the C3-Cloud system. With continuous testing activities it ensures the quality and correct implementation of the various system components. WP9 develops training material and organizes training workshops for the technology trial in the 3 pilot sites. Preparation and monitoring of pilot site activities for and during the piloting trial is ensured through the development of a research protocol and a monitoring tool. The C3-Cloud system will be evaluated regarding usability, acceptability and user satisfaction. Data on usage of healthcare resources will be obtained during the trial to enable the modelling of C3-Cloud impact and plan up-scaling scenarios for the pilot sites. The evidence from the trials, analysed through this work package, is essential to help demonstrate the value of C3-Cloud’s approach and solutions, to justify wider adoption across Europe.

The WP9 activities are laid out in response to the overall hypothesis of the C3-Cloud system that: “The use of the C3-Cloud application by the multi-disciplinary team and patients together with their informal caregivers improves the management of multi-morbid patients with poly-pharmacy”.  Based on this hypothesis, the study objective was defined as §to evaluate the user-centred design, the usefulness, usability, feasibility and user experience of the C3-Cloud pilot application among patients, their informal caregivers and MDT members”. For this, a number of C3-Cloud application outcome propositions will be monitored and the C3-Cloud impact will be modelled for large-scale deployment of the C3-Cloud application.

 The leading research question for the evaluation of the pilot technology trial of the C3-Cloud application is therefore: “Is the use of a personalised ICT tool that facilitates coordinated care planning, treatment optimisation and patient self-management acceptable to patients with multiple long-term conditions and their team of health professionals?”


Achievements to date

We have defined a clear research question and an approach that is able to predict and model the impact of the application when implemented at scale. In addition, criteria for patient and clinician selection were defined in collaborative work with WP10. A recruitment tool to randomize patients to the different evaluation layers and intervention and control groups was developed. Procedures for obtaining informed user consents were defined. This was followed by a period in which the research protocol was written and the application testing and usability testing protocols were defined. A 4-layered evaluation approach was defined and questionnaires for each evaluation layer were developed and translated to Spanish and Swedish. Application and C3-Cloud system components, as well as initial usability testing was performed in collaboration with the three pilot sites, Inserm, Warwick University and empirica. A training strategy and plan common for the three pilot sites were developed by Kronikgune with minor local deviations for each pilot site. A codebook summarizing all evaluation means was developed and shared with the pilots and their ethics commissions. All evaluation questionnaires for patients and members of the multi-disciplinary teams were drafted on an online platform for testing and later use during the trial period. The large-scale impact modelling was prepared by defining impact factors for the modelling and setting an initial predictive model (using discrete event simulation) for testing and validation.

Current status

The current and upcoming activities focus on the training of all C3-Cloud system users, finalizing preparations for the technology trial and monitoring the progress in the pilot sites. All users will be involved in thorough evaluation activities by answering surveys and by obtaining anonymous data on their healthcare resource usage. During the final project months, WP9 will have increased workload around analyzing the evaluation results and modelling the up-scaling potential of the C3-Cloud system, taking a systemic perspective.



Intended outcomes

We aim to provide in our final deliverable D9.6 exploratory insights into the usability, user acceptability and user satisfaction with the C3-Cloud system and related training materials. In addition we will present a model for the impact of C3-Cloud on resource consumption and involved stakeholders by combining a discrete event simulation with a systemic cost-benefit assessment.


• D9.1 Functional and Non-functional Testing Criteria for C3-Cloud Components
D9.2 Qualitative and Quantitative Evaluation Criteria for C3-Cloud Pilot Application
D9.3 Test and Evaluation Report for C3-Cloud Components 
D9.4 User Training Materials and Preparation Outcomes 
• D9.5 Interim Evaluation Results for the C3-Cloud Pilot Application 
• D9.6 Final Result of Evaluation and Modelling Large-Scale Impact of the C3-Cloud Pilot Application 

Those marked with a  will be publicly available after approval.