Study Report

Preparation before testing:

The project aimed to evaluate the user-friendliness of two technical solutions. Both solutions have developed new functionalities beyond their previously approved indications. Health personnel, patients, and next of kin participated in the test, and informed consent was planned for all participant groups. Patients with two or more chronic conditions were considered a vulnerable population; therefore, ethical aspects were carefully considered when preparing the test protocol.

A test protocol was developed based on the MDCG 2024-3 Guidance on the content of the clinical investigation plan for clinical investigations of medical devices from the EU. The project engaged in dialogue with the Medical Products Agencies in all three countries, and the user-friendliness testing was considered to fall under MDR Article 62(1), as the manufacturers intend to CE-mark the solutions.

Informed consent for health personnel, patients and next of kin were written. Two questionnaires were included for the participants to evaluate the technical solutions after testing. System usability score (SUS), a validated form to test the subjective assessment of usability was used. In addition, a project-based questionnaire for functionality was developed.  Comprehensive Investigator Brochures (IB) were developed for both solutions.

We applied to Medical Product Agency, and regional/national ethical boards in all three countries where testing was to be performed. The process of preparing documents was time consuming. After approval was received, participants were invited to test the solutions.

All three test sites invited user organizations to participate in the testing. The level of interest varied, and in some countries several user organizations had to be contacted to recruit enough participants. The allocation of solutions to participants was random, although not formally randomized. We aimed to include participants with two or more chronic conditions. Through dialogue with user organizations and potential participants, we found that the definition of a chronic disease was not always clear to them. As a result, some individuals were uncertain whether they met the inclusion criteria. Participants were therefore asked to self-assess whether they fulfilled the medical definition of having two chronic conditions. The project did not collect any information about participants’ specific diagnoses.

In retrospect, we recognize that the inclusion criteria could have been less stringent, which might have facilitated communication and improved understanding among potential participants. At this stage of development, where the primary aim was to assess user friendliness, including participants with any chronic condition would likely have been sufficient.

All participants were invited to the test sites for in-person testing. We did not offer the option to complete testing at home or remotely for any participant group.

Conducting the testing

The testing was conducted individually. The solutions were tested stand alone with no integration with other EHR systems. Participants received a brief introduction to the project and signed the consent form before testing began. Patients and next of kin tested only one solution, and each session lasted approximately 1.5 hours. Clinicians tested both solutions, which took about 3 hours.

Participants had varying levels of prior experience with health technology, and several therefore required guidance during testing. Study personnel were present and assisted participants with any challenges that arose.

The suppliers provided test user accounts for all participants, with relevant content tailored to the test tasks. Medical devices and smartphones with the applications installed were distributed before testing. The suppliers were available to provide technical support throughout the sessions.

We encountered some technical issues during testing. Some were resolved immediately by the supplier, while others remained unresolved, preventing completion of certain tasks. These unresolved issues may have influenced how participants rated the usability afterwards.

Results for the testing

A total of 150 participants were included.