Skip to main content

Patients’ experience using an app for home remote monitoring of heart failure for a university hospital in Quebec, Canada

Abstract

Background

The use of mobile health apps for remote monitoring has considerable potential for patient care. However, more research is needed to gather patients’ experience with their use. This study evaluates heart failure patients’ experience of the effects of remote monitoring on the management of heart failure (HF) using a smartphone app, the challenges faced in such use, and patients’ suggestions for improving the efficacy of this device.

Method

A qualitative study was conducted with all patients who had used the TakeCare™ app for at least three months during the exploratory phase of the Continuum project. Patients were recruited before COVID-19 began, and they used the app during this period. The app allows patients to submit their daily vital signs and answer questions about their symptoms to help the clinical professionals adjust their treatment plan as needed. Individual interviews were conducted by telephone or videoconference using a semi-structured individual interview guide. All interviews were recorded, transcribed and analyzed using the thematic analysis method.

Results

Of the 29 patients invited, 5 had since died, 10 did not respond, two gave positive feedback on the app but declined the interview, and 12 patients agreed to participate in an individual interview. The participants all liked the app’s ease of use. They particularly praised the contribution it made to monitoring their disease, both by themselves and by clinicians. The app also allowed them to stay connected to their clinical team and have better access to healthcare professionals. Patients suggested, as an improvement, that the feedback and communication features should be personalized to make them more individualized and interactive.

Conclusion

The TakeCare™ smartphone app was well received by patients with heart failure. Its use could facilitate better remote monitoring of heart conditions in the home as well as improve access to clinical teams.

Peer Review reports

Introduction

Heart failure (HF) is a frequent, serious and complex disease [1,2,3]. HF is a leading cause of mortality and morbidity in elderly people and the first cause of hospitalization among people aged 65 and over in Canada [4, 5]. One in five patients is readmitted to hospital within 30 days of discharge due to either multiple decompensations or associated severe comorbidities [6, 7]. However, nearly 40% of these hospital readmissions could be avoided by implementing an appropriate clinical management program [7, 8]. Despite therapeutic advances and Canadian guidelines for the management of HF [9], early readmission (< 30 days after hospital discharge) of HF patients remains a challenge for healthcare systems in Canada, particularly in Quebec, where patients are treated in public healthcare facilities [10, 11]. Given the resources provided by the current healthcare system, it is difficult to provide all patients suffering from HF with personalized follow-up in a specialized setting after discharge. To remedy this situation, it has been proposed that healthcare teams and patients be provided with new tools that would enhance remote patient monitoring [12,13,14].

For the past few decades, as a result of improved information and communication technologies (ICT) used in healthcare, remote monitoring has enabled patients to provide information to their healthcare professionals for improved accessibility and continuity of HF management. These new technologies include smartphones and their apps. Several studies have shown gains from remote monitoring through the use of a mobile app by reducing the hospital readmission rate [15, 16], improving patient self-management [17,18,19,20,21], and ensuring health-related quality of life [19]. However, qualitative studies have shown that there are still several difficulties to be overcome in terms of the adoption of mobile phone apps and patient adherence to this type of intervention [17, 19, 22, 23]. The main obstacles impeding the adoption of these devices are poor integration of the technology into the patient’s daily life and technological difficulties experienced during use [24,25,26,27]. Also, studies need to be carried out to assess the patient experience with remote monitoring apps. They would make it possible to better understand how patients interact with this innovative method of follow-up given their circumstances, daily life, level of knowledge, expectations and objectives. Across all types of healthcare, patients should ideally be playing an active role in the management of their illness and as active as possible a role in the continuous management of their health conditions [23, 28,29,30].

In a context characterized by limited clinical resources and technology development in Quebec, in 2019, the Quebec University of Montreal Hospital Center (CHUM) implemented the Continuum project, intended to optimize the quality and continuity of follow-up for HF patients in an effort to reduce hospital readmissions [31]. To carry out this project, an exploratory study was conducted, consisting of two phases. The first phase involved exploring the feasibility of the Continuum web platform in a pilot study, using the TakeCare™ mobile app for remote monitoring of HF patients in the home. A qualitative study designed to analyze the patients’ perceptions was embedded in this phase. Patients were therefore recruited from December 2019 to February 2020, before the start of the COVID-19 pandemic, and they used the app during the first months of the pandemic when regular in-person visits were limited as much as possible under national guidelines [32]. The second phase consisted of a randomized controlled study [33]. The TakeCare™ mobile app allows patients to submit their vital signs and symptoms on a daily basis and have their questions answered, with the treatment plan being adjusted as required. The purpose of this app is to facilitate monitoring of the disease and patient autonomy and self-care.

After they gave their consent, patients were invited to download the TakeCare™ mobile app onto their mobile phone or electronic tablet. They were also given the choice of whether to use the connected objects (weight scale, blood pressure monitor and glucometer), but could refuse some and/or continue to use their own if they preferred. During the follow-up period, through the app, manually or via Bluetooth, the patients transmitted the following data daily: (1) their vital signs (body temperature, weight, blood pressure, heart rate, blood sugar), (2) their level of physical activity (the number of steps taken in the last 24 h, counted by Google Fit), and (3) answers to a questionnaire on symptoms related to HF (shortness of breath, ankle swelling, fatigue) (Fig. 1).

Fig. 1
figure 1

The TakeCare™ app for heart failure home monitoring

The data is sent to the hospital for monitoring and, if necessary, to modify the treatment plan, which is automatically generated by the platform. A clinical nurse takes care of daily remote data monitoring (five days a week) and the mobile app generates follow-up alerts as required. When one of the values ​​exceeds previously set thresholds, an alarm is triggered. The nurse communicates with the patient by telephone to assess the situation and provide instructions, if necessary. Each patient is followed for a period of three to six months, to optimize patient follow-up after hospitalization [9].

The objectives of this qualitative study are to assess: (1) the effects of the TakeCare™ smartphone app on the home management of heart failure (HF), (2) the challenges encountered in using the app, and (3) the patient’s suggestions for improving patient acceptance of this device. The CHUM research ethics committee approved the study (reference number: 19.364).

Materials and methods

Design and population

A qualitative study was conducted using interviews [34]. Patients who had used the TakeCare™ mobile app in the pilot phase of the Continuum project for at least three months were invited to take part in an individual interview.

Selection criteria

For the pilot study, patients were recruited from the hospitalized population at the CHUM and among those followed at the CHUM’s HF clinic, from December 2019 to February 2020. The inclusion criteria were as follows: being at least 18 years of age, having a confirmed diagnosis of HF with New York Heart Association (NYHA) class II or higher symptoms, owning an electronic tablet or smartphone and being able to understand and use the technology (a research nurse verified that the last two criteria had been satisfied). The exclusion criteria were as follows: having another serious illness (such as end-stage renal failure), having had a kidney or liver transplant, having cancer for which the prognosis is one year or less, suffering from severe cognitive disorders, or not having been followed for 12 consecutive weeks. All patients in the pilot study were invited to participate in the qualitative study.

Recruitment

All patients in the pilot study were invited by telephone and email by the research team. Of the 29 patients invited, 5 had since died, 10 did not respond, and 2 gave positive feedback on the app but refused to participate in the individual interviews. Ultimately, 12 patients gave their consent and participated in our qualitative study.

Recruitment for our qualitative study was carried out from February 25 to March 8, 2022, two years after the pilot phase.

Data collection

Qualitative data was collected through individual semi-structured interviews conducted by two rechearchers (MPP and TXHV) [34]. The interview guide was developed based on the research objectives and the literature. An initial version of the guide was tested in a pilot study involving three patients with HF. We wanted to test the patients’ understanding of the questions and their relevance. This led to the development of the final version used in the study (Table 1). Each interview was conducted in French by telephone or videoconference. They were all recorded. The most relevant quotes were transcribed and translated into English to illustrate the results. The source of each quote is indicated by an “I,” for “interview,” followed by a number according to the person who spoke.

Table 1 Semi-structured interview guide used with patients with HF

Data analysis

The interview data were analyzed using a thematic analysis method [34]. The first stage involved one researcher (TXHV) transcribing all the interviews and the two co-researchers (TXHV and MPP) were directly involved in coding them and reading the interviews to familiarize themselves with the data. In the second stage, the two co-researchers independently coded three interviews in order to bring out various themes using an inductive approach [35]. During the third stage, two meetings were held of the co-researchers and research professionals to draw up a codebook with 20 main coding categories (Table 2) [36]. Once the codebook had been stabilized, the fourth stage involved the two co-researchers coding all the interviews using QDA Miner software (version 6.0.2.). In the fifth stage, the codebook was further refined, and the final stage involved making sense of the results according to the research objectives [35] and selecting the most relevant quotes to illustrate the themes. The quotes were then translated into English. The methodology was reported using the COREQ checklist of consolidated criteria for qualitative research [37].

Table 2 Themes, subthemes and categories of items

Results

Of the 14 patients contacted by telephone and invited to participate, 2 gave positive feedback but declined to participate and 12 gave their consent and participated in the interview, of whom 9 were male. Five of these nine men were over 69 years of age (range: 50 – 81), and one of the three women was over 63 years (range: 60 – 64) (Table 3). Four out of 12 participants had an NYHA score of 3 or 4. The distribution of sex, age group and NYHA score does not differ between the interviewed group and the non-interviewed group, which consisted of patients remaining in the original patient list (Chi2 test p-values > 0.05). Data saturation was reached after 10 interviews. The interviews lasted 10 to 45 min.

Table 3 Demographic characteristics of the interviewees (n = 12)

Perceived benefits of takecare™

Expected benefits when introducing takecare™

When clinicians asked patients to use the app, some had no particular expectations regarding its use. Most of the other patients valued the idea of being monitored as a way to improve the safety of their care. Patients perceived that the TakeCare™ app could help them monitor their vital signs and symptoms, better intervene in their health condition and maintain contact with the attending staff.

Useful data for self-monitoring the disease

Patients are interested in tracking data about their health status. They like having information on the evolution of their disease in real time: graphs, histories, and weekly summaries:

“It helped me in the sense that I was weighing myself almost every day, I know my pressure. I have statistical data. If it increases a lot, I try to find out why. Yes, it forces me to be more vigilant." (I3)

Patients paid more attention to changes in their physical signs and life habits. The app helped patients adhere to their treatment.

“The app allows me to actually see my situation. There was a point, like for a month, when I knew that my weight was tending to increase a little. So I thought there were a few things popping up, and I started paying attention. It’s up to me to reduce it when it needs to be reduced." (I10)

Benefits of remote monitoring

Four of 12 patients felt that the telemonitoring system was like having “a safety net.” They had the impression that their data was being used appropriately by the nurses and doctors, such that they could always be in contact with a health professional for support in managing their disease and, therefore, take better control of their situation. They particularly valued the calls from clinicians when an alert signal was triggered. They believe that by using the data entered into TakeCare™, physicians can recognize abnormal signs promptly and adjust their medication, based on the impact of the medications on their daily life.

“I used to take 11 pills, and now I only take 5. Yes, they stopped a lot of the medications. They told me I didn't need some of them anymore." (I6)

Data entered into TakeCare™ is used in a variety of ways. The data is transferred directly to the hospital and the healthcare staff through a secure web-based dashboard. Moreover, patients can present their data to a medical specialist in other departments during consultations.

“When I meet with my specialist doctor who takes care of my diabetes, he weighs me, and I tell him ‘I have data, everything is entered in TakeCare, and we can go and see it.’ These are the actual curves that I have for my disease, for diabetes. In fact, everything is there, everything is already entered on the CHUM's IT platform, so it's wonderful." (I7)

"You know, it's complicated in hospitals nowadays. They’re short-staffed, ... But with TakeCare, it's really effective." (I7)

Feeling of safety

All participants mentioned the importance of daily follow-ups through the Internet. If the data deviated from the norm, the clinical team contacted them by telephone for an appropriate response. Participants reported that they found this to be reassuring and felt it created a safe environment:

“I find it’s good to have nurses who can call me to tell me how it works. Since my hospital stay, I have felt really privileged, and I already feel that I have good follow-up. I live in a safe haven, actually." (I10)

Improved health status

Finally, participants observed an improved health status, weight loss, stabilization of their blood sugar, a better ability to walk, to eat, etc.

“I am more stable, whereas before I was not stable at all." (I3)

"It has already helped me a lot to know the state of my health, and then the data were getting better and better, so I saw that my health was getting better." (I8)

Expected benefits: why patients want to use TakeCare™ over the long term

The four main reasons why patients are interested in continuing to use TakeCare™ are: (1) to continue receiving the perceived benefits of being followed up over the longer term (I3, 6, 7, 10); (2) to be able to self-monitor their physical signs over time (I1, 5, 6, 8, 9, 10); (3) to maintain a connection with the attending staff in order to make appointments (I7), and (4) to have easy access to care (I7).

Ease of use and difficulties

All the patients found TakeCare™ very easy to use. None of them reported having problems installing the app on their mobile phone. Two patients out of 12 (patients 5 and 11) needed from a few days to a week to become familiar with the measurement activities required during the day and enter their data. Once this became part of their routine, it would take them 5 to 10 min to complete all the measurements of physical signs and enter the data. Regardless of how the daily data was entered ─ either automatically via Bluetooth between the app and the connected measuring devices (the weight scale, the blood pressure monitor, the capillary glucometer) or manually ─ it was not a problem for them.

“It’s not difficult. In fact, it’s practical. Access to the site is easy, and all the links on the site are also easy to use." (I10)

“Before, we would take our weight, diabetes and blood pressure, and it was programmed, so that it was entered directly. Now you have to enter the data manually. But that doesn’t matter, it’s nothing." (I7)

On the other hand, the participants reported some problems with device connectivity and the software (the weight scale, the blood pressure monitor).

"It's a bit annoying, because sometimes the devices don't connect to Bluetooth, so you have to disconnect everything and then reconnect." (I1)

“I use Google to count the number of steps. Google does the math, but TakeCare doesn’t communicate with Google. I don't know why, but it never works. But hey, it's not really necessary." (I6).

Patient suggestions for improving the effectiveness of the TakeCare™ app

Improve some technical aspects of TakeCare™

Patients have suggested various improvements to make it possible to enter more data using the app. They proposed entering data from an SpO2 pulse oximeter, or being able to follow their psychological state through validated scales.

“Maybe we could have psychological questions, if possible. Because people are often depressed when this type of diagnosis [HF] is made, and some counselling from a psychologist is also important. Basically, when someone is sick, the doctor takes care of the biological aspects, an endocrinologist takes care of the balance sheet, … but it’s also necessary to deal with psychological issues." (I5)

Moreover, according to the patients, some minor technical aspects need to be improved to make the app even more practical and useful, such as adding some options on the time period in the charts (for a week, a month or certain months…), allowing the results to be printed, giving occasional or continuous access to another person:

“I would have appreciated being able to invite him to review my results or send them to him. It would be nice if we could invite people to access our results, so that they could follow the data continuously or only from time to time." (I1)

Personalize information in messages and emails

Each week, patients receive a reminder on their phone inviting them to enter the data in the app, and then they receive a summary of the data by email. The participants reported that the messages are too brief and therefore they are not personalized for each patient. They do not feel that they are being followed by the nursing staff, but rather by a computer program. In addition, they noted that the summary only presents statistical data for the week; it lacks interpretation, remarks on changes, or advice or recommendations to help them better respond. When their physical signs and symptoms are stable, they found that the data summaries become monotonous and repetitive. Patients become less attentive and, ultimately, less motivated to consult the data.

“I read it all the first time, but not after that! I thought I would always see the same message, in the same place. It may help if there were little reminders saying that there is new information in the email. So we're going to go and look, because we're curious, and it would be helpful. " (I3).

Therefore, they recommend more interactive communication by SMS, email or telephone. For example, they proposed using SMS messages to inform them of a new email, provide recommendations based on the results or their health status, and give them links to health information, such as on self-monitoring and self-management of HF and associated diseases.

The participants suggested that emails should include a statistical summary of their physical signs, along with some interpretation. In addition, they thought advice should be provided to help them monitor their data or manage their disease.

“I have a follow-up at the end of the week that gives me my average blood pressure, how many times I measured my blood sugar and the change in my weight. But I would also like to be provided with targets, so that I can improve." (I2)

"Perhaps when we send emails, it would be appropriate to indicate where we are, to provide advice, recommendations and even information on our disease to help us on this subject." (I3)

Add more interactive modes of communication

The app is mainly used to transmit clinical data to clinicians. However, patients reported that they are also interested in communicating proactively with their clinical team. They like to be able to communicate other worrisome signs and discuss their concerns or their treatment plan. The patients made some suggestions for better communication:

  • Add a free text field for patients to share concerns with their clinician and get a reliable and specific answer from the clinical team:

    “Instead of having only 4-5 closed questions, perhaps have an open question with a free field in which to write, not a huge space, but to say, for example, ‘I reduced my dose of furosemide.” If that worries the nurse, she will call me." (I1)

    “It might be a good idea to be able to send a quick question to an HF team using the TakeCare app.” (I5)

  • Have different codes that would allow patients to indicate the reason for their call to the clinical team, such as: an urgent case, a request for clinical or therapeutic information, a request for more information on their statistical follow-up results, a request for psychological information, the need to speak to a another patient, etc.

    “For example, when I'm waiting at the clinic, or when I don't know the person who is taking care of me in the emergency room. So if I had something like a reply, an immediate response on TakeCare, I could do it. I think that would be very good.“ (I5)

  • Discussion forum for patients:

    “I want to be proactive about my health, and for that I need to talk to someone on the phone. I’m also interested in chatting with people who have the same disease as me.“ (I2)

General comments

All participants indicated that they thought the TakeCare™ app could help patients with HF and should be used as early as possible in the patients’ pathway.

“It’s significant that people like me, with HF, really need the services. Something technology-like that is as important as a new drug. It can also improve life. So unclog the system as soon as possible.“ (I7).

Discussion

The results demonstrate that HF patients were positive about the contribution made by the home-monitoring app. In particular, they mentioned the app’s contribution to monitoring their disease, both by the clinicians and themselves. The app is easy to use in daily life. The patients emphasized the importance of developing interactive communication and personalizing the feedback received from the clinical team to facilitate self-care. These experiences and suggestions create potential avenues for further interventions and research on the development of mobile apps for patients with complex chronic pathologies.

In this section, we will discuss the three main lessons learned from the patient experience: (1) the technical specifics of the app; (2) the benefits of the app for increasing the capacity of patients and health professionals to manage the disease; and (3) the development of interactive communication between patients and their care team. Lastly, we will discuss the limitations of this study and provide some recommendations.

Technical specifics

The simplicity of use of any technology, along with its benefits, are two essential aspects of all models of acceptance and adoption of new technologies in general and of mobile apps in particular [38]. Previous studies have shown that the main obstacles to the use of mobile apps are a poor previous experience when using technical tools and poor integration of the technology into the patients’ daily lives [24,25,26,27]. In our study, all participants said that this app is simple and easy to use. First, this may be because the participants we initially recruited were already able to use a smartphone, so they had no trouble using this app. In other words, the app is easy to use for people with basic smart device skills. All of our participants indicated that they found the app pragmatic in terms of installing it and using its interface and functionalities. Its ease of use and the availability of technical support enabled the app to be effectively adopted in such a way that the patients integrated it into their lifestyle [39].

The technical difficulties noted by our participants consisted of connectivity problems with Bluetooth and the app’s measurement objects and data entry. For some patients, this issue sometimes interrupted daily data entry [23]. However, in this study, even if patients had trouble connecting measuring devices, they had no problem entering their data manually. Additionally, most patients continued to enter data manually after the pilot phase had ended.

Lastly, the patients suggested improving certain technical functions, such as by adding graphs, being able to print data or providing another person with occasional or continuous access. These minor changes could indeed make use of this device’s data more user-friendly.

Benefits of the app: simultaneous monitoring by the clinician and the patient

Riegel and Dickson's chronic disease self-care theory [40, 41] presents the concept of self-care in three parts: self-care maintenance (e.g., adherence to self-care behaviours such as regular exercise and taking medication as prescribed), self-care monitoring (e.g., regular measurement of changes, routine testing), and self-care management (e.g., changing the diet or medication dose based on the detection and interpretation of symptoms). In this sense, the TakeCare™ app promotes self-care by encouraging patients to monitor the occurrence of symptoms of HF and diabetes on a daily basis (self-care monitoring) and to adhere to treatment, by making them aware of changes in their symptoms (self-care maintenance), but the aspects of the app that need improvement relate to self-care management of the disease. Patients said that they needed information on how to interpret their data (symptom monitoring) and on how to intervene to better manage their disease and makes changes to their lifestyle (self-care management). They would prefer if the summary data was accompanied by interpretation, advice and recommendations. These proposals value personalizing the data in its interpretations and recommendations in order to make the device more user-friendly, increase motivation and promote self-care in relation to the disease.

Personalized health information

Each patient is unique due to their specific needs and experiences. Providing information tailored to the needs of each patient is a particularly important factor in ensuring the usefulness of mobile health apps [24, 42,43,44,45,46,47,48]. This is all the more true in this instance, since HF is a complex disease and is often associated with many other diseases [49]. How can patients be assisted in their efforts to find relevant and useful information on their specific case?

The literature mentions several ways of providing interactive information that helps place the focus on patient needs. For example, rather than providing information intended for all patients, Foster incorporated short self-assessment questionnaires matched with automated feedback that was tailored to patients’ responses to encourage them and guide them toward accessing the appropriate resources [50]. A systematic review of articles by Vo et al. showed that appropriate supplemental information empowers patients and helps them manage their disease, take responsibility and adhere to treatment [25]. According to Dickson, health information should enable the development of tactical and situational skills [51]. Tactical skills are related to strategies, such as adherence to prescribed treatments and/or medications, and situational skills relate to strategies such as how to decide whether or not to take an extra dose of a diuretic. Such a decision-making support system is currently being reviewed and developed in connection with the European HeartCycle FP7 project, in which the system guides most management decisions and does not ask for help from other people, only an expert in relatively rare cases [52]. In summary, the automated interactive functionality of the app is a tool designed to help patients take an active role and help them make decisions autonomously in particular clinical situations, while reducing the healthcare providers’ workloads [53].

Like most remote monitoring apps, TakeCare™ has a health information feature. However, few patients mentioned the benefits of this feature. Since providing information is an essential condition to encouraging patients to take responsibility and take charge of their disease [42], further studies should be conducted on this subject to know which aspects need to be improved in order to optimize the use of TakeCare™.

Interactive exchanges: the optimal way to support patients

In Quebec, the heath care system is based on a patient-healthcare partnership. Patients are encouraged to play the role of a partner and member of the clinical team to ensure adequate care and follow-up of their health issue [54]. To achieve this, interaction and communication between patients and the entire clinical team must be given special attention in the services provided by the healthcare system [43]. The ease of communication between patients and healthcare providers made possible by mobile health would appear to be one of its great promises. Several suggestions were presented to improve the nature of communication between patients and their care team and between the patients themselves.

Direct communication between patients and clinicians

The majority of participants suggested the introduction of phone calls in order to benefit from direct interactions with the clinical team. This initiative may include: (1) an occasional call from a team member (a real person) to inquire about their physical and psychological health, (2) a method for calling in the event of an emergency [55], (3) a call method linked to the telephone number of the attending staff member, or a default SMS message such as “I need to speak to you, please call me” [50], (4) a free text field to share their concerns with their clinician and benefit from a reliable and specific response from the clinical team [56]. Son et al. pointed out that if participants can communicate with their clinical team at any time, illness-related anxiety will be reduced [48]. Thus, this type of communication is more human and reliable, but it can increase the clinical team’s workload. Development of the necessary functionalities should take into consideration their added value and resource availability.

Interaction between patients

In addition to their interactions with healthcare providers, participants suggested functions that would allow for conversations with other people who have had or are going through a similar experience (other patients, caregivers or a patient-guide) via discussion forums. In particular, in Quebec, a new approach based on an optimal partnership between patients and health professionals is being deployed by mobilized patient volunteers called “accompanying patients” (AP) who are integrated into clinical teams [57]. These volunteer patients have acquired knowledge from their own experience living with the disease and are willing to share this information with other patients receiving treatment. First, the emotional and informational support offered by APs improves the patient care experience. Second, the participation of APs in the activities of the clinical team facilitates communication between patients and the health staff. Interactive exchanges between patients, or with accompanying patients who have faced similar situations, are therefore one of the most useful aspects of therapeutic patient education [57,58,59].

In order to develop the app's interaction features and content in a manner appropriate to the needs, it is strongly recommended that they are co-constructed by app designers, clinicians and patients, including the elderly [60].

Specific context and limitations

This study was conducted with all the patients in the intervention group of the pilot project phase, with the exception of those who could not be reached by telephone. Nevertheless, many limitations should be considered when discussing the results. The first limitation is tied to the characteristics of the patients included in the study. By selecting only patients who had participated in the pilot phase, our sample consisted of individuals who had been using the app for three months. This means that we were unable to identify any difficulties encountered by those individuals who had stopped using the app beforehand. An additional potential selection bias is the fact that, despite reaching saturation with the data collected, the participants in the individual interviews were likely to be more capable of using the app than those who did not show up for the interview (deceased patients or patients who could not be contacted). Second, there is a potential recall bias, including difficulties experienced at the start of app use by the participants due to the timing of the data collection, which was conducted two years after the end of the pilot phase. Third, it should be recalled that our study was carried out during the COVID-19 pandemic, a period when health restrictions led to encouraging remote appointments [32]. In this context, patients’ experience and acceptance of the TakeCare™ mobile app for home monitoring may be overestimated. Although we still conduct follow-ups with telehealth today (something that did not exist before COVID), the context of this study limits any interpretations of how patients would evaluate it today in the absence of a health crisis. Fourth, due to the context of the project (pilot phase), and the organizational context of the site under review (a single location, CHUM), the results obtained must be interpreted accordingly. The next study will explore the points of view of other end users, such as clinicians and hospital managers, to gain an overall view of user experiences.

Recommendations

Recommendations for using the app

The use of mobile apps for HF monitoring by HF patients holds promise for improving patient access to remote care, particularly in patients at a high risk of decompensation. The perception of the app’s ease of use in daily life could help predict its acceptance and continued use over time. It is essential to develop features for interactive communication with the care team or between patients as well as personalized health information features to maintain motivation and optimize the capacity of patient self-care. Furthermore, the tracking of psychological scales and perceived health in HF-focused mobile apps remain to be explored. The adoption of this new technology by elderly patients could be promoted through encouragement by the clinical team and IT support. The distribution of a monitoring app should not be limited to the hospital and its network; it must be made accessible to cardiologists, endocrinologists and other family physicians in the region, so that patients are better supported in the management of their health problems and care plans.

Future research

Based on the results of this study, we are suggesting some potential avenues for future research on how to improve the self-management information features of the TakeCare™ app. In addition, we would need qualitative studies aimed at exploring the experience and suggestions from the various actors concerned, such as clinicians, managers, decision-makers, programmers and IT equipment suppliers, in order to optimize and update this innovation. It would also be relevant to conduct studies to measure the impact on patients’ quality of life, mental health and relationship with their clinical team.

Conclusion

Patients are essential partners of their care team. Patients with HF aged 50 years and older and with many co-occurring diseases may be keen to use a mobile app if it is user-friendly. This strengthens the relationship between patients and the nursing staff, with practical impacts on the safety of not only their physical health but also their psychological health. As part of this pilot study, patients suggested making improvements to promote communication through the app and personalized messages to help them become more autonomous in their self-care of HF and associated chronic diseases. The mobilization and participation of patients in partnership with clinicians from various disciplines, app designers and app suppliers are strongly recommended in the creation, development, implementation and updating of such apps. Randomized clinical trials currently being conducted to evaluate the medico-economic impact of innovative interventions related to the TakeCare™ app among cardiac patients will confirm the effectiveness of the Continuum platform and this new app.

Availability of data and materials

The data used in this study is stored and anonymized. The data is not publicly available, but it may be available upon formal request (please contact the corresponding author).

References

  1. Savarese G, Lund LH. Global public health burden of heart failure. Card Fail Rev 2017;3:7—11. 2017;3(1). https://doi.org/10.15420/cfr.2016:25:2.

  2. Son YJ, Lee Y, Lee HJ. Effectiveness of Mobile Phone-Based Interventions for Improving Health Outcomes in Patients with Chronic Heart Failure: A Systematic Review and Meta-Analysis. Int J Environ Res Public Health. 2020;17(5). https://doi.org/10.3390/ijerph17051749.

  3. Fondation des maladies du cœur et de l’AVC du Canada. Des effets insuffisants. Le combat contre l'insuffisance cardiaque sur tous les fronts. Toronto (Canada); 2022. https://heartstrokeprod.azureedge.net/-/media/pdf-files/canada/2022-heart-month/cavc-rapport-insuffisance-cardiaque-2022-final.ashx?la=fr-ca&rev=e8266d7bbddc47fe81ac8ec5c2111db6.

  4. Butrous H, Hummel SL. Heart Failure in Older Adults. Can J Cardiol. 2016;32(9):1140–7. https://doi.org/10.1016/j.cjca.2016.05.005.

    Article  PubMed  Google Scholar 

  5. Information CIfH. Hospital stays in Canada 2023 [updated December 22, 2023. Available from: https://www.cihi.ca/en/hospital-stays-in-canada.

  6. Canadian Institute for Health Information. All-Cause Readmission to Acute Care and Return to the Emergency Department. Ontario: Health System Performance; 2012 2–05–2022]. Available from: https://publications.gc.ca/collections/collection_2013/icis-cihi/H118-93-2012-eng.pdf.

  7. Phelan D, Smyth L, Ryder M, Murphy N, O’Loughlin C, Conlon C, et al. Can we reduce preventable heart failure readmissions in patients enrolled in a Disease Management Programme? Ir J Med Sci. 2009;178(2):167–71. https://doi.org/10.1007/s11845-009-0332-6.

    Article  CAS  PubMed  Google Scholar 

  8. Goldgrab D, Balakumaran K, Kim MJ, Tabtabai SR. Updates in heart failure 30-day readmission prevention. Heart Fail Rev. 2019;24(2):177–87. https://doi.org/10.1007/s10741-018-9754-4.

    Article  CAS  PubMed  Google Scholar 

  9. Ezekowitz JA, O'Meara E, McDonald MA, al. e. 2017 Comprehensive Update of the Canadian Cardiovascular Society Guidelines for the Management of Heart Failure. Canadian Journal of Cardiology. 2017;33(11):1342–433. https://doi.org/10.1016/j.cjca.2017.08.022.

  10. Ministère de la Santé et de Services Sociaux. MED-ÉCHO — Rapports statistiques périodiques 2009–2010. Québec; 2016. https://publications.msss.gouv.qc.ca/msss/document-001792/.

  11. Poon S, Leis B, Lambert L, MacFarlane K, Anderson K, Blais C, et al. The state of heart failure care in Canada: minimal improvement in readmissions over time despite an increased number of evidence-based therapies. CJC open. 2022;4(8):667–75. https://doi.org/10.1016/j.cjco.2022.04.011.

    Article  PubMed  PubMed Central  Google Scholar 

  12. McDonagh TA, Metra M, Adamo M, Gardner RS, Baumbach A, Group ESD, et al. 2021 ESC Guidelines for the diagnosis and treatment of acute and chronic heart failure: Developed by the Task Force for the diagnosis and treatment of acute and chronic heart failure of the European Society of Cardiology (ESC) With the special contribution of the Heart Failure Association (HFA) of the ESC. Eur Heart Journal. 2021;42(36):3599–726. https://doi.org/10.1093/eurheartj/ehab368.

  13. Desnos M, Jourdain P. Télémédecine : une solution d’avenir pour l’insuffisance cardiaque ? Bull Acad Natl Med. 2020;204(8):817–25. https://doi.org/10.1016/j.banm.2020.07.034.

    Article  Google Scholar 

  14. Bashi N, Karunanithi M, Fatehi F, Ding H, Walters D. Remote Monitoring of Patients With Heart Failure: An Overview of Systematic Reviews. J Med Internet Res. 2017;19(1):14. https://doi.org/10.2196/jmir.6571.

    Article  Google Scholar 

  15. Park C, Otobo E, Ullman J, Rogers J, Fasihuddin F, Garg S, et al. Impact on readmission reduction among heart failure patients using digital health monitoring: feasibility and adoptability study. JMIR Med Inform. 2019;7(4):e13353. https://doi.org/10.2196/13353.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Suthipong C. The Feasibility of the Heart Failure Mobile Phone Aide Application (Android Apps.) on Self-Management Measured by Improved Physical Function and Reduced Hospital Admissions Related to Heart Failure in Thailand. Washington DC: The Catholic University of America; 2018. https://islandora.wrlc.org/islandora/object/cuislandora%3A213555.

  17. Athilingam P, Jenkins B, Johansson M, Labrador M. A Mobile health intervention to improve self-care in patients with heart failure: pilot randomized control trial. JMIR Cardio. 2017;1(2):e3. https://doi.org/10.2196/cardio.7848.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Radhakrishnan K, Toprac P, O’Hair M, Bias R, Kim MT, Bradley P, et al. Interactive Digital e-Health Game for Heart Failure Self-Management: A Feasibility Study. Games Health J. 2016;5(6):366–74. https://doi.org/10.1089/g4h.2016.0038.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Wei KS, Ibrahim NE, Kumar AA, Jena S, Chew V, Depa M, et al. Habits Heart App for Patient Engagement in Heart Failure Management: Pilot Feasibility Randomized Trial. JMIR Mhealth Uhealth. 2021;9(1):e19465. https://doi.org/10.2196/19465.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Wonggom P, Nolan P, Clark RA, Barry T, Burdeniuk C, Nesbitt K, et al. Effectiveness of an avatar educational application for improving heart failure patients’ knowledge and self-care behaviors: A pragmatic randomized controlled trial. J Adv Nursing. 2020;76(9):2401–15. https://doi.org/10.1111/jan.14414.

    Article  Google Scholar 

  21. Yanicelli LM, Goy CB, González VDC, Palacios GN, Martínez EC, Herrera MC. Non-invasive home telemonitoring system for heart failure patients: A randomized clinical trial. J Telemed Telecare. 2021;27(9):553–61. https://doi.org/10.1177/1357633X19899261.

    Article  PubMed  Google Scholar 

  22. Middlemass JB, Vos J, Siriwardena AN. Perceptions on use of home telemonitoring in patients with long term conditions – concordance with the Health Information Technology Acceptance Model: a qualitative collective case study. BMC Med Inf Decis Making. 2017;17(1):1–13. https://doi.org/10.1186/s12911-017-0486-5.

    Article  Google Scholar 

  23. Sanders C, Rogers A, Bowen R, Bower P, Hirani S, Cartwright M, et al. Exploring barriers to participation and adoption of telehealth and telecare within the Whole System Demonstrator trial: a qualitative study. BMC Health Serv Res. 2012;12(1):220. https://doi.org/10.1186/1472-6963-12-220.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Schmaderer M, Miller JN, Mollard E. Experiences of Using a Self-management Mobile App Among Individuals With Heart Failure: Qualitative Study. JMIR Nursing. 2021;4(3). https://doi.org/10.2196/28139.

  25. Vo V, Auroy L, Sarradon-Eck A. Patients’ Perceptions of mHealth apps: meta-ethnographic review of qualitative studies. JMIR Mhealth Uhealth. 2019;7(7):e13817. https://doi.org/10.2196/13817.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Woods LS, Duff J, Roehrer E, Walker K, Cummings E. Patients’ Experiences of Using a Consumer mHealth app for self-management of heart failure: mixed-methods study. JMIR Hum Factors. 2019;6(2):e13009. https://doi.org/10.2196/13009.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Seto E, Leonard KJ, Cafazzo JA, Barnsley J, Masino C, Ross HJ. Perceptions and Experiences of Heart Failure Patients and Clinicians on the Use of Mobile Phone-Based Telemonitoring. J Med Internet Res. 2012;14(1:e25):10. https://doi.org/10.2196/jmir.1912.

  28. Jones L, Grech C. The patient experience of remote telemonitoring for heart failure in the rural setting: a literature review. Contemp Nurse. 2016;52(2–3):230–43. https://doi.org/10.1080/10376178.2015.1111154.

    Article  PubMed  Google Scholar 

  29. Jouet E, Flora L, Gino, Las Vergnas O. Construction et reconnaissance des savoirs expérientiels des patients. Note de synthèse. Pratiques de Formation - Analyses. 2010;2010(58–59):83. https://hal.science/hal-00645113.

  30. Wolf JA, Niederhauser V, Marshburn D, LaVela SL. Definition of Patient Experience Reexamined to Acknowledge the Human Experience in Healthcare. Patient Exp J. 2021;8(1):16–29. https://doi.org/10.35680/2372-0247.1594.

  31. Greybox. Transformer la prise en charge des patients atteints d’insuffisance cardiaque grâce aux technologies numériques Montreal (Québec): Greybox; 2019 [Available from: https://www.greybox.ca/fr/2019/12/20/transformer-la-prise-en-charge-des-patients-atteints-dinsuffisance-cardiaque-grace-aux-technologies-numeriques/.

  32. Canadian Cardiovascular Society. Guidance from the CCS COVID-19 rapid reponse team - Ambulatory management and diagnostic testing during the COVID-19 crisis 2020 [Available from: https://ccs.ca/wp-content/uploads/2021/02/CCS_Guidance_for_Ambulatory_and_Diagnostic_Testing.pdf.

  33. Tournoux F. Continuum: Digital Health as Part of the Care Trajectory of Heart Failure Outpatients. ClinicalTrialsgov [Internet]. 2022. Available from: https://www.clinicaltrials.gov/ct2/show/NCT05377190.

  34. Bryman A. Social research methods. 5th ed. Oxford, United Kingdom: Oxford University Press; 2016.

    Google Scholar 

  35. Greg Guest, Kathleen M. MacQueen, Namey EE. Themes and codes. 2012 2022/04/28 [cited 28/04/2022]. In: Applied Thematic Analysis [Internet]. Thousand Oaks, California: SAGE Publications, Inc., [cited 28/04/2022]; [32]. Available from: https://methods.sagepub.com/book/applied-thematic-analysis.

  36. Boyatzis R. Transforming Qualitative Information: Thematic Analysis and Code Development. 1998.

  37. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57. https://doi.org/10.1093/intqhc/mzm042.

    Article  PubMed  Google Scholar 

  38. Bin Azhar FA, Dhillon JS, editors. A systematic review of factors influencing the effective use of mHealth apps for self-care. 2016 3rd International Conference on Computer and Information Sciences (ICCOINS); 2016 15–17 Aug. 2016. Kuala Lumpur (Malaysia): IEEE; 2016. https://doi.org/10.1109/ICCOINS.2016.7783213.

  39. Ware P, Dorai M, Ross HJ, Cafazzo JA, Laporte A, Boodoo C, et al. Patient adherence to a mobile phone-based heart failure telemonitoring program: a longitudinal mixed-methods study. JMIR Mhealth Uhealth. 2019;7(2):e13259. https://doi.org/10.2196/13259.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Riegel B, Jaarsma T, Lee CS, Strömberg A. Integrating Symptoms Into the Middle-Range Theory of Self-Care of Chronic Illness. ANS Adv Nurs Sci. 2019;42(3):206–15. https://doi.org/10.1097/ans.0000000000000237.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Riegel B, Dickson VV. A situation-specific theory of heart failure self-care. J Cardiovasc Nurs. 2008;23(3):190–6. https://doi.org/10.1097/01.Jcn.0000305091.35259.85.

    Article  PubMed  Google Scholar 

  42. Wali S, Demers C, Shah H, Wali H, Lim D, Naik N, et al. Evaluation of heart failure apps to promote self-care: systematic app search. JMIR Mhealth Uhealth. 2019;7(11):e13173. https://doi.org/10.2196/13173.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Brunner-La Rocca H-P, Fleischhacker L, Golubnitschaja O, Heemskerk F, Helms T, Hoedemakers T, et al. Challenges in personalised management of chronic diseases—heart failure as prominent example to advance the care process. EPMA Journal. 2016;7(1):1–9. https://doi.org/10.1186/s13167-016-0051-9.

    Article  Google Scholar 

  44. Birkhoff SD, Smeltzer SC. Perceptions of Smartphone User-Centered Mobile Health Tracking Apps Across Various Chronic Illness Populations: An Integrative Review. J Nurs Scholar. 2017;49(4):371–8. https://doi.org/10.1111/jnu.12298.

    Article  Google Scholar 

  45. Coorey GM, Neubeck L, Mulley J, Redfern J. Effectiveness, acceptability and usefulness of mobile applications for cardiovascular disease self-management: Systematic review with meta-synthesis of quantitative and qualitative data. Eur J Prev Cardiol. 2018;25(5):505–21. https://doi.org/10.1177/2047487317750913.

    Article  PubMed  Google Scholar 

  46. Scott IA, Scuffham P, Gupta D, Harch TM, Borchi J, Richards B. Going digital: a narrative overview of the effects, quality and utility of mobile apps in chronic disease self-management. Aust Health Rev. 2020;44(1):62–82. https://doi.org/10.1071/AH18064.

    Article  PubMed  Google Scholar 

  47. BezerraGiordan L, Tong HL, Atherton JJ, Ronto R, Chau J, Kaye D, et al. The use of mobile apps for heart failure self-management: systematic review of experimental and qualitative studies. JMIR cardio. 2022;6(1):e33839. https://doi.org/10.2196/33839.

    Article  Google Scholar 

  48. Son Y-J, Oh S, Kim EY. Patients’ needs and perspectives for using mobile phone interventions to improve heart failure self-care: A qualitative study2020 2 June 2020; 76(9):[2380–90 pp.].https://onlinelibrary.wiley.com/doi/abs/10.1111/jan.14455.

  49. Burbach BE, Cohen MZ, Zimmerman LM, Schmaderer MS, Struwe LA, Pozehl B, et al. Post-hospitalization transition to home: patient perspectives of a personalized approach. J Nurs Educ. 2016;6(5):32 https://www.sciedupress.com/journal/index.php/jnep/article/view/8283.

    Google Scholar 

  50. Foster M. A Mobile Application for Patients With Heart Failure: Theory- and Evidence-Based Design and Testing. CIN: Computers, Informatics. Nursing. 2018;36(11):10 https://journals.lww.com/cinjournal/Fulltext/2018/11000/A_Mobile_Application_for_Patients_With_Heart.6.aspx.

    Google Scholar 

  51. Dickson VV, Riegel B. Are we teaching what patients need to know? Building skills in heart failure self-care. Heart Lung. 2009;38(3):253–61. https://doi.org/10.1016/j.hrtlng.2008.12.001.

    Article  PubMed  Google Scholar 

  52. Vera-Muñoz C, Arredondo MT, Ottaviano M, Salvi D, Stut W, editors. HeartCycle: User interaction and patient education. 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); 2013 3–7 July 2013. https://doi.org/10.1109/EMBC.2013.6611166.

  53. Dierckx R, Pellicori P, Cleland JGF, Clark AL. Telemonitoring in heart failure: Big Brother watching over you. Heart Fail Rev. 2015;20(1):107–16. https://doi.org/10.1007/s10741-014-9449-4.

    Article  CAS  PubMed  Google Scholar 

  54. Pomey M-P, Dumez V, Boivin A, Rouly G, Lebel P, Berkesse A, et al. The Participation of Patients and Relatives in Quebec’s Health System: The Montréal Model. 1 ed. Marie-Pascale P, Jean-Louis D, Vincent D, editors. Quebec: Palgrave Macmillan Cham; 2019. 17–61. https://link.springer.com/book/10.1007/978-3-030-14101-1.

  55. Hälleberg Nyman M, Frank C, Langius-Eklöf A, Blomberg K, Sundberg K, Wengström Y. Patients’ perspective on participation in care with or without the support of a smartphone app during radiotherapy for prostate cancer: qualitative study. JMIR Mhealth Uhealth. 2017;5(7):e107.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Husted GR, Weis J, Teilmann G, Castensøe-Seidenfaden P. Exploring the Influence of a Smartphone App (Young with Diabetes) on Young People’s Self-Management: Qualitative Study. JMIR Mhealth Uhealth. 2018;6(2):e43.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Pomey M-P, Menear M, Drouin C, Saba T, Roy D. Amélioration des soins et des services en première ligne pour la gestion des maladies chroniques : le programme COMPAS+ au Québec. Revue française des affaires sociales. 2020;1:325–35. https://doi.org/10.3917/rfas.201.0325.

    Article  Google Scholar 

  58. Schnall R, Bakken S, Rojas M, Travers J, Carballo-Dieguez A. mHealth technology as a persuasive tool for treatment, care and management of persons living with HIV. AIDS Behav. 2015;19(2):81–9. https://doi.org/10.1007/s10461-014-0984-8.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Shorey S, Yang YY, Dennis C-L. A Mobile Health App-Based Postnatal Educational Program (Home-but not Alone): Descriptive Qualitative Study. J Med Internet Res. 2018;20(4): e119. https://doi.org/10.2196/jmir.9188.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Srinivas P, Cornet V, Holden R. Human factors analysis, design, and evaluation of Engage, a consumer health IT application for geriatric heart failure self-care. Int J Human Comp Interact. 2017;33(4):298–312. https://doi.org/10.1080/10447318.2016.1265784.

    Article  Google Scholar 

Download references

Acknowledgements

We are deeply grateful to the patients who completed the survey for their time and openness to sharing their experiences. We would also like to thank to Jim Kroening for its translation.

Funding

M.‐P. P. has a Senior Career Award financed by the Quebec Health Research Fund (FRQS), the Centre de Recherche du Centre Hospitalier de l'Université de Montréal and the Ministère de la Santé et des Services sociaux du Québec. This work was supported by a joint grant from Medteq + (innovation for health) and Prompt (#10–26), including financial contributions from Boehringer Ingelheim (Canada) Ltd./ ltée, Greybox Solutions Inc. and Mitacs. Medteq + , Prompt and the other contributors played no role in the design and conduct of the study; in the extraction, management, analysis, or interpretation of the data; or in the preparation, review or approval of the manuscript. BI was provided with a copy of the manuscript for review for medical accuracy. The content reflects the views of the authors only.”

Author information

Authors and Affiliations

Authors

Contributions

Conceptualization, F.T. and M-P.P.; methodology, M-P.P.; formal analysis, investigation, TXH.V., M-P.P., P.R., LJ.C., E.R., S.G, S.B., J.N., P.L., E.M-T., F.T.; writing-original draft preparation, TXH. V., M-P.P., E.R., S.G,; review, TXH.V., M-P.P., P.R., LJ.C., E.R., S.G, S.B., J.N., P.L., E.M-T., F.T.; project administration, F.T.. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Marie-Pascale Pomey.

Ethics declarations

Ethics approval and consent to participate

This study was conducted in accordance with the Declaration of Helsinki and approved by the Centre hospitalier de l’Université de Montréal’s research ethics board (REB). All participants were required to sign an informed consent form approved by the Centre hospitalier de l’Université de Montréal’s REB. Project ID: 19.364.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vo, T.X.H., Pomey, MP., Charles, L.J. et al. Patients’ experience using an app for home remote monitoring of heart failure for a university hospital in Quebec, Canada. BMC Digit Health 2, 68 (2024). https://doi.org/10.1186/s44247-024-00126-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s44247-024-00126-4

Keywords