Development of the Compassionate Technology Scale for Professionals (CTS-P): value driven evaluation of digital mental health interventions

Abstract

Background

Compassion is an essential and beneficial value in mental healthcare. However, how digital mental health interventions influence compassion in treatment has not been systematically investigated, due to the lack of appropriate measurement instruments. To address this gap, we developed the Compassionate Technology Scale for Professionals (CTS-P), aimed at mental health professionals.

Methods

We used Q-methodology, a method that combines quantitative and qualitative analysis to explore shared viewpoints on a particular topic, to select and refine items. Participants were 15 professionals from different areas of mental healthcare. In individual sessions, while thinking aloud, they sorted 35 statements on relevance for evaluating technology use on compassion. The statements were based on a scientific conceptualization of compassion with five elements. The sorting task was followed by a short interview to explore participants’ associations with compassion and technology.

Results

With by-person factor analysis, we found three different viewpoints among participants, all with eigenvalues > 1 and with a total explained variance of 63.02%. We selected prioritized items of each viewpoint and for each theoretical element of compassion, resulting in a full scale (15 items) and a short scale (3 items). Based on qualitative input from participants, the scale was adapted to clarify its focus and the wording of items. While thinking aloud, participants shared benefits and critical notes regarding technology and compassion.

Discussion

Together with key stakeholders, we developed the CTS-P. Most participants prioritized the potential of technology to obtain more information and be closer to their client in facilitating compassion. The main critical note participants had was that technology is not necessary to support compassion at the therapist’s experiential level. This emphasizes the need to further explore how mental health professionals and technology can complement each other in a system of compassionate care. Future research should explore the factor structure, validity, and reliability of the scale through psychometric validation.

Conclusions

The CTS-P can make the extent to which technology for mental healthcare influences elements of compassion measurable and comparable. The scale can prove useful for prioritizing the value of compassion in both the development and use of digital mental health interventions.

Compassion is considered an essential value in (mental) healthcare [1,2,3]. In short, compassion refers to the awareness of suffering and the motivation to act to alleviate it [1, 4, 5]. The presence of compassion in care has been shown to have multiple beneficial effects for both clients and health professionals. If care is perceived as compassionate (by the client or professional), this can improve client satisfaction, treatment outcomes, job satisfaction of clinicians, and improve communication between the client and healthcare professional [6, 7]. Clients and professionals in mental healthcare highly value personal contact and the presence of compassion in care [8, 9].

Currently, with rising care demands and costs, technology is considered a potential contributor to sustainable mental healthcare. However, technology is also seen as a threat to personal, compassionate contact by researchers and practitioners [10, 11]. Consequently, it becomes crucial to assess how the use of technology in treatment is experienced in terms of compassion. A tool to evaluate the use of technology in mental healthcare on compassion does not yet exist, but is called for [11, 12]. Therefore, the current paper describes the development of a scale to make compassion in treatment with technology visible and measurable, as well as an exploration of the perceived links between technology and compassion in mental healthcare. It starts from the perspective of mental healthcare professionals and combines a quantitative and qualitative approach referred to as ‘Q methodology’ [13].

Defining compassion

Compassion has been described in different ways [14]: as an emotion [15], from a religious perspective [16], from an evolutionary perspective [17, 18], as a skill [19], a motivational caring system [18], or as a virtue [20]. All of these approaches have merits, illustrating the comprehensive and versatile nature of compassion. However, there is no common consensus on how to define compassion. Therefore, Strauss et al. [1] reviewed and synthesized various definitions and conceptualizations of compassion with the aim of improving the ability to measure the construct. Strauss et al. proposed a definition with five elements:

“1) Recognizing suffering; 2) Understanding the universality of suffering in human experience; 3) Feeling empathy for the person suffering and connecting with the distress (emotional resonance); 4) Tolerating uncomfortable feelings aroused in response to the suffering person (e.g. distress, anger, fear) so remaining open to and accepting of the person suffering; and 5) Motivation to act/acting to alleviate suffering” [1].

In this comprehensive definition, compassion is seen as a cognitive, affective and behavioral process. For our current purposes, this definition has several advantages. First, it is an overarching synthesis of previous definitions and conceptualizations from different perspectives, so that all major descriptions of compassion are combined. Second, it was constructed specifically to advance the understanding and measurement of the construct of compassion, which fits our current goals. Third, viewing compassion as a process rather than merely a human emotion opens doors to exploring how technology might support compassion within the context of mental healthcare, resonating with the viewpoint proposed by Morrow and colleagues. In their review linking AI to compassion, Morrow et al. advocated viewing compassion as a motivational caring system, in which both humans and technology can play a role [21]. Fourth, the definition proposed by Strauss et al. [1] has previously been shown to be helpful for connecting compassion to technology in systematic scoping reviews [12, 21].

Compassion and digital mental health technologies

Currently, technological advancements are making their way into mental healthcare. Examples of digital mental health interventions (DMHIs) include online treatment modules, serious games, virtual reality, social robots and conversational agents [12]. In combination with ‘traditional’ face to face treatment, they can form blended treatments [22]. If used well, and if delivered with some form of human support, DMHIs have been proven to be as effective as ‘traditional’ face-to-face treatment [23,24,25,26,27]. However, mental healthcare professionals are still hesitant to structurally integrate DMHIs in their daily practice and afraid that DMHIs hinder the therapeutic alliance [28, 29]. The addition of DMHIs to treatment introduces different types of communication and treatment possibilities compared to traditional face-to-face treatment, and it is unclear how this influences compassion between a client and therapist. Unfortunately, existing scales for evaluating technology typically focus on usability, acceptance and effectiveness [30,31,32] and do not involve compassion [11, 33]. Although high scores on usability metrics are also desirable, they do not necessarily correlate with high scores on therapeutic constructs [34].

Current literature has mostly focused on the effects of digital (self-)compassion interventions on a person’s (self-)compassion (e.g. [35]). By contrast, perspectives on the relationships between DMHIs and compassionate care are limited and mixed. While an earlier study reported opportunities for DMHIs to facilitate compassion, it also raised concerns about technology detracting from the compassionate relationship [11]. A systematic scoping review on compassion and technology for mental healthcare found several potential ways in which the process of compassion can be supported by DMHIs [12]. In the review, three ‘compassionate roles’ for technology in mental healthcare are described: technology showing compassion to a person, enhancing self-compassion in a person (i.e. compassion-based interventions), or facilitating compassion between people. The review also provides examples of how technologies in each role can foster elements of compassion; for example by helping to keep track of mood and symptoms to support the recognition and understanding of suffering. However, the included DMHIs were not evaluated on whether they were also experienced as contributing to compassion by their users. To evaluate whether and to what extent different DMHIs actually foster compassion, a standardized measurement instrument is needed.

Current objectives and approach

In the current paper, we describe the development of a scale to evaluate the degree to which a technology facilitates compassion between the client and the professional, starting from the perspective of the professional. The aim is to develop a scale with both a full version and a short form, so that it is suitable for different research designs. For example, the full version could be used in a more extensive survey to evaluate a series of blended treatment sessions or for a pre-post research design. The short form would be fitting for a repeated measures design, for example after each treatment session or after an interaction with a DMHI.

Compassion is a multifaceted construct, which has never been related to technology before in the form of a scale. Therefore, we used an explorative yet structured study design that allows us to collect direct input from the intended users of this new scale, referred to as Q-methodology [36]. This approach is especially helpful if the studied topic is not easily defined or understood, as is the case here. Our primary objective is to derive a relevant and comprehensive compassionate technology evaluation scale that reflects the perspectives of professionals in mental healthcare. This scale should be useful in guiding future assessments and implementations of digital mental health interventions. Therefore, our research questions are as follows:

• 1. Which items should be selected to represent different clusters of mental health professionals and the theoretical elements of compassion, both for a full scale and a short form?

• 2. What associations and interpretations do participants have when reviewing the items regarding the use of technology to support compassion in mental healthcare?

• 3. Based on the selected items and qualitative feedback from participants, what is a suitable full and short scale for mental health professionals to evaluate DMHI use on compassion?

Methods

Design

In Q-methodology, participants sort statements into a normal distribution according to their subjective judgment on a given variable (e.g. level of attractiveness or importance) while thinking out loud, followed by a short interview [36]. As such, Q-sorting allows for the collection of both qualitative and quantitative data and is helpful in providing a rich picture of participants’ understanding and opinions on a topic [13]. A form of factor analysis is used to analyze the ‘Q-sorts’, in which each individual’s sort is correlated with all other sorts. The factors identified through the analysis represent clusters of participants who sorted statements similarly [37].

As Q-sorting provides insight into clusters of participants who sorted items similarly, each of their viewpoints can be represented in the selection of scale items, answering research question 1. To answer research question 2, the think-aloud data and the short interviews after the sorting task give insight into the associations and interpretations participants had when reviewing the items. To answer research question 3 and compose the final scale, we integrated the selected items with participants’ qualitative input from the think-aloud data and interviews. The Q-sort method is viewed as an attractive type of data collection by participants [36], potentially because participants are asked to think freely about what they find important in evaluating a certain topic, and can be more actively engaged.

Participants and setting

This study was conducted with 16 mental health professionals, recruited from three different suborganizations of one overarching Dutch mental healthcare organization. Participants were recruited using convenience sampling from the networks of the researchers, and approached via e-mail with information on the study. The three foundations involved in this study offer different types of mental healthcare: the first one provides primary mental healthcare, the second one provides secondary mental healthcare, and the third provides primary and secondary mental healthcare fully online. In primary mental healthcare, mild to moderate mental health problems are treated, while in secondary mental healthcare more complex and serious psychiatric disorders are treated. Involving professionals from these different types of mental healthcare gave us a more representative sample of the Dutch mental healthcare system. We recruited professionals that were employed at the organization at the time of the study, and who were fluent in Dutch. Among the 16 participants, one did not complete the quantitative part, citing the forced-choice rank-ordering's near-normal distribution as too restrictive for expressing their opinions. Therefore, we only analyzed the qualitative data for this participant and not their (incomplete) quantitative data. For another participant, the recording of the Q-sort meeting failed for technical reasons. For this participant, we only analyzed their quantitative data. Therefore, for both the quantitative and the qualitative data we ended up with 15 participants. Table 1 shows the demographic information of the participants of whom we analyzed the quantitative and qualitative data.

Table 1 Demographic information of the participants (n = 15)

Materials

Item construction

To develop the potential items for the scale, preparatory work was done drawing from the conceptualization and item development phases of scale development described by Alber et al. [38]. We started with desk research on compassion, existing compassion scales, and technology evaluations. Next, we selected a suitable definition of compassion [1] to apply to technology. As preliminary work, a focus group with 6 mental health professionals (a different sample from the current one) was held to explore their opinions about DMHIs, blended treatment, and compassion. The focus group showed that compassion was indeed an important value to mental health professionals, and that they preferred offering blended treatment over providing either physical or digital treatment alone. Subsequently, a systematic scoping review was conducted to provide an overview of links between compassion and DMHIs in current scientific literature [12]. Based on the previous activities, a first set of 35 items to evaluate DMHI use on compassion was generated and a feedback session was held with 6 experts from the research fields of compassion, eHealth development and scale development. Based on the feedback session, the items were refined.

The five elements of compassion as proposed by Strauss et al. [1] formed the basis for our set of statements, or the ‘Q-sample’. These five elements were all represented equally, with 6 items for each compassion element. Additionally, 5 overarching items were added for a short form of the scale. These items did not refer to elements of compassion, but to compassion itself, for a more general assessment. All items referred to a ‘platform’ used between a client and professional in mental healthcare. By using ‘platform’ to describe forms of digital mental health interventions (DMHIs), we aimed to use a simple and succinct word that could refer to different types of DMHIs. The items were presented in Dutch, as our participants were Dutch-speaking.

The ideal number of items used in a Q-sort study depends on the complexity of the subject [39]. All possible aspects of the research topic must be represented. Typically Q-samples consist of 20–50 statements [36]. In our study, we used 35 statements, to maintain a balance between doing justice to the comprehensive construct of compassion and not making the sorting task too time-consuming for our participants.

Q-sort materials

The Q-sort task consisted of two parts (see Fig. 1). For the first part, a basic distribution with three different columns was used. In the second part, a near-normal distribution was presented, with two items in each of the extremes, gradually increasing to seven items in the middle pile.

Fig. 1

Schematic display of the first (left) and the second part of the Q-sort (right)

The Q-sort study was conducted online using Microsoft Teams and the free website ‘Qsortware’ [40], which allows participants to be invited via email to complete the Q-sort in their browser. The shiny application ‘QMethod’ [41] was used as a resource to analyze the Q-sort data, as well as Atlas.ti (version 24.0.0.29576) for Windows for qualitative analysis.

Procedure

The Q-sorts took place in online one-to-one sessions that lasted an average of 49 min (std. dev. 9 min, min. 31 min, max. 1 h and 5 min). The sessions were recorded (screen and audio). All collected data were anonymized so that they could not be traced to the participants. Participants provided written informed consent before taking part in the study.

At the start of each session, the researcher provided background information on blended treatment (combining eMental health and face-to-face treatment [22]), as well as compassion and its elements, the aim of the current study and the envisioned scale. Next, participants were invited to the Qsortware website and asked to share their screen via Microsoft Teams. Then, the first part of the sorting task was explained to participants, emphasizing that there were no right or wrong answers, and that our interest lay in the personal opinion of the participant. Therefore, participants were asked to think out loud during the whole sorting task. If a participant had been quiet for a while during sorting, they were asked to share what they were considering and how they made their sorting choices.

In the first part, participants sorted the Q-sample (35 items) into three different columns (see Fig. 1, left) based on the following question: “how important are these items for evaluating whether technology supports compassion in treatment, according to you?”. The three columns had the following headings: ‘least important to measure’, ‘neutral’, and ‘most important to measure’. Above these three columns, the items were presented one after another, so that participants could drag them to one of the three columns and move them around as desired. The items were shuffled across compassion elements, but the order of the items was not randomized across participants, as the online program that we used did not allow for this. After all the items were sorted in the three columns and participants were satisfied with their distribution, they could continue to the second part of the sorting task.

In the second part, participants were shown a near-normal distribution (see Fig. 1, right). Above the empty Q-sort distribution, the three columns from the first part were shown, containing the previously sorted items. Now, participants were asked to distribute their previously sorted items into the near-normal distribution of the empty Q-sort. Again, participants were free to keep moving the items around until all positions were filled and they were fully satisfied with the resulting distribution.

When participants were finished with sorting the items into the distribution, the Qsortware website was closed and a short semi-structured interview followed. The interviews followed an interview guide that was developed for the current study, and can be found in Additional file 1. Participants were asked what they thought of the task, what they thought of the items, how they made decisions during sorting, and whether any items were superfluous or missing according to them.

Data analysis

Quantitative data

The Q-method is an inverted technique of factor analysis [36]: the resulting data matrix has respondents as variables and items as cases. Thus, respondents are correlated instead of items, and factors represent clusters of participants with similar Q-sorts. The aim of the statistical analysis of the Q-sorts is to determine which distinct clusters represent shared viewpoints on the topic.

The quantitative data from the Q-sorts were analyzed with by-person factor analysis. First, a correlation matrix of all Q-sorts was produced, indicating the degree of correspondence between respondents. Then, we used Principal Component Analysis (PCA) with varimax rotation and automatic flagging of significant sorts (loadings) on each extracted factor. The analysis also identified confounding respondents (loading significantly on > 1 factor) and non-significant respondents (not loading significantly on any factor). To find a factor model with a number of clusters that fit our data well, we examined whether all eigenvalues were greater than 1 (in line with [42]). Furthermore, we checked whether at least two Q-sorts loaded significantly on each factor and not on any other factor (in line with [43]), meaning that at least two Q-sorts had correlations of at least 0.5 with ‘their’ factor, and cross-loadings with other factors below 0.40 (in line with [44]). Next, we checked whether the clusters were interpretable in a substantive manner by checking whether we could find meaningful differences in viewpoints between the clusters. Based on the identified clusters, we selected the items that each cluster prioritized on average as most important.

Qualitative data

The similarities and differences in prioritized items between the clusters were examined and interpreted to make a meaningful choice about the number of factors that fit our data, as well as to describe these clusters.

The full Q-sort sessions were transcribed verbatim, consisting of thinking out loud data and the short interviews after the sorting task. Thematic analysis was used to analyze this data following the guidelines of Braun and Clarke [45]. Thus, relevant features of the data were systematically coded by one coder (CvL), generating initial codes. These codes were merged into potential themes. Separately from this bottom-up approach, the five elements of compassion were coded in a top-down manner, forming the theme “compassion”. All themes were then reviewed and a thematic map of the analysis was created. Next, the bottom-up themes were further defined and named. Using this coding scheme, a second coder (JA) also coded 13% of the data, with a percentage agreement of 75% with the first coder (CvL), which was deemed satisfactory. Differences in coding were discussed until consensus was reached, and two new codes were added (“Less active client with technology use” and “Integration of technology in workflow”). The first coder (CvL) then checked all the data again with the two added codes. Finally, the report was produced with a selection of illustrative quotes to help answer our research questions.

Results

Research question 1: clusters and prioritized items

A model with three factors had a good fit for the data, with all eigenvalues greater than 1, and a total explained variance of 63.02%. Each of these factors had at least two Q-sorts that correlated more than 0.5 with it, and less than 0.4 with other factors. The factors represent three different clusters of participants who sorted the statements similarly. Of the 15 participants, none loaded significantly on more than one factor, so there were no confounding sorts. However, one participant did not fit in this model, because the participant did not load significantly on any of the factors. While clusters 1 and 2 correlated strongly (r = 0.55), cluster 3 did not correlate with cluster 1 (r = 0.03) or cluster 2 (r = 0.00). These correlations show that there is some consensus among the first two clusters, while the third factor represents an alternative perspective. A representative Q-sort was made for each of the three clusters. The scores of these representative Q-sorts can be found in Table 2.

Table 2 Representative item scores per cluster, from least (-4) to most important (4). Selected items are in bold

Based on the three-factor model, we selected items for the scale (see Table 2, in bold). For each of the five elements of compassion, the item that was sorted as most important by each of the three clusters was selected. The Q-sample also included more general, overarching items about compassion. From these items, the ones that were sorted highest in each cluster were selected for the short form of the scale. The factor loadings for each Q-sort and factor, Q-sorts flagged in each factor, item z-scores and distinctive and consensus statements are reported in Additional File 2.

Below, we will describe the three clusters, using the collected qualitative data in the theme ‘compassion’ (see Table 3) to illustrate the accompanying perspectives. It is important to keep in mind that although we recognized different trends in these clusters, participants showed a large range of nuanced ideas and sometimes changed their minds during a sorting session. The clusters presented below are based on their final thought-through Q-sort. The items we refer to can be found in Table 2 based on their number.

Table 3 ‘Compassion’ theme and codes

Cluster 1: ‘Goal-oriented’ (n = 8), factor eigenvalue 4.76; explained variance 31.73%

For this largest cluster of participants, it was most important that technology supports knowing how a client is doing and working towards goals (item nr. 11 and 8), or doing what is necessary to alleviate suffering (item nr. 6 and 25). As one participant put it: “The goal in mental healthcare is of course to help a client and to meet the client’s own request for help. […] So if a platform can contribute to setting healthy goals with the client, that is one of the most important things in treatment, I think” (pp23). The least important aspect of using technology compassionately was whether it supports the awareness of the universality of suffering (item nr. 15 and 34). That was not a role for technology according to this group: “I think it is important that we realize that we are people and that we can go through problems. I just don't think it's important that the platform does that. I think the therapist needs to do that themselves, in a different way” (pp18). Technology also does not necessarily need to support the therapist in dealing with their own difficult emotions (distress tolerance, item nr. 9 and 20). The role of technology in supporting (cognitive) empathy was seen as more neutral or ambiguous: “Some clients fill out a diary with photos and then I understand much better what a client means. That does help to gain more understanding. But I think if you need a platform to understand a client, I don’t think that’s okay” (pp14).

Cluster 2: ‘Client-centered’ (n = 4), factor eigenvalue 3.06; explained variance 20.42%

For this cluster, it was also highly important that technology supports knowing how a client is doing in order to learn as much as possible about the client (item nr. 8 and 16), but in this case so that it can facilitate empathy for the client (item nr. 3 and 17). One participant explained this priority as follows: “If I am more informed, then I also know what is needed. And if I know what's going on, I can also have more empathy than if I don't know what's going on” (pp1). Key aspects include being aware of the client's well-being, empathizing with the client's experience, and showing understanding for it. Alleviating suffering and working towards goals was seen as more neutral, sometimes described as less fitting with compassion: “I find it important, but I don’t think it says the most about how compassionate that technology is” (pp9). Even less important was whether technology supports awareness of the universality of suffering: “’The platform helps not to look down on a client’s suffering’, no, I don’t do that anyway. A platform doesn’t have much to do with that” (pp5). The least important aspect was whether technology assists the therapist in dealing with their own difficult emotions (item nr. 20 and 31). This was experienced as something a therapist should be able to manage as a prerequisite for their job; otherwise, one cannot be a therapist: “Protecting yourself in therapy is very important, but I wonder whether you need a medium for that. In principle I hope that doesn’t matter” (pp11). If necessary, therapists can address this with colleagues during intervision according to participants.

Cluster 3: ‘Empathy and Self-Care’ (n = 2), factor eigenvalue 1.63; explained variance 10.87%

The most crucial aspect for this cluster was that technology supports affective empathy (feeling for another’s suffering, item nr. 14 and 17). Affective empathy was followed by technology aiding therapists in dealing with their own difficult emotions and not looking down on the client (item nr. 2, 9, and 20): “Sometimes it’s good to just read something, then you can step back, give a response and think: oh, I can refine that. Because at a certain point you can be completely overwhelmed, and no longer watch with distance. So that may be important” (pp10). Noticing the client's suffering and working towards goals were seen as more neutral. Supporting the awareness that suffering is universally human was not important according to this group: “That would mean that I am not sufficiently aware of that. I don’t have that idea, so this would rather irritate me” (pp22). Cognitive empathy (facilitating understanding of the client and comprehending their problems), was the least important for evaluating technology use on compassion, according to this group (item nr. 4 and 30). One participant explained: “As far as I am concerned, understanding is more about comprehension. And comprehension is more on the cognitive side than the affective side” (pp22). This quote implies that this cluster viewed compassion more as an affective process instead of (also) a cognitive one.

Research question 2: therapists’ associations with items

During the Q-sort sessions, the participants were asked to think aloud and were subsequently interviewed. This provided many interesting remarks about the reasoning behind their sorting choices and their views on DMHIs and compassion. These remarks could be divided into three main themes (see Table 4): Benefits or possibilities of technology and compassion, Critical or nuancing notes on technology and compassion, and Role divisions and ways of working, which we will each describe below, along with a selection of illustrative quotes. We also linked the codes within these themes to the ‘compassion’ theme (Table 3) where overlap was found. For an overview of illustrative quotes for each code, see Additional File 3.

Table 4 Themes and codes for participants’ associations with items on compassion and technology

Benefits or possibilities of technology and compassion (121 quotes)

By far the most prevalent code within this theme (72 occurrences) was ‘Technology helps to get more information about the client or to be closer to the client’. Most often (31 times), this benefit was mentioned when participants discussed items related to recognizing suffering: “I think it is a kind of low threshold accessibility [with technology]. If someone is not feeling well I am more likely to notice this with the support of such a platform than without” (pp1). This benefit was also often linked to empathy (22 times): “What I get back from clients is that they like the fact that they feel like we are very close to them, and that we are also warm in contact, while actually we do that via video calling or chat or contact in other ways” (pp4).

The code ‘Technology gives structure or facilitates goal-oriented working’ was assigned 16 times, and was exclusively linked to the compassion element of alleviating suffering, with participants explaining that the structured form of DMHIs can help to remain ‘to the point’. Another benefit was found in the code “Technology gives a helpful distance” (13 occurrences). This benefit was almost exclusively linked to the compassion element of distress tolerance. Furthermore, the code ‘Technology that supports the professional themselves’ was found 12 times, and was also mostly linked to distress tolerance (5 times). Some professionals spoke about the potential of a module aimed at therapists, to help them remain compassionate. The final benefit that was mentioned was ‘Technology can decrease stigma’ (8 occurrences), which was often linked to understanding the universality of suffering (5 times): “I find the modules in [eHealth provider] very accessible and human. They are made in such a way that when you work with them, you think, yes, this could just as easily have happened to me. It is very little ‘care provider language’, so not so stigmatizing” (pp14).

Critical or nuancing notes on technology and compassion (119 quotes)

Participants in the Q-sort not only mentioned benefits and possibilities, but also made critical or nuancing remarks on linking technology to compassion. By far the most prominent code here was ‘Technology is not necessary for elements of compassion’ (103 occurrences). This code was most often linked to understanding the universality of suffering (36 times): “For the client, I think it would be good if a certain platform, or perhaps video messages from former clients would allow clients to feel less stigma about certain complaints or mental suffering, but I think for myself that that is not really an issue for me” (pp15). This code was also often linked to distress tolerance (34 occurrences), with participants explaining that they also did not feel that they would need to use the technological platform as a buffer for distress. Similarly, the code was linked to empathy (19 occurrences), which some participants did not feel that they needed a platform for. A few times, the code was linked to alleviating suffering (10 occurrences): “I think it’s important to set good goals and with the platform you can work on that, but it can also be done in other ways” (pp21).

Another critical note made by participants was ‘Less or less accurate information about the client via technology’ (10 occurrences), which was always linked to recognizing suffering. A code that was mentioned a few times (6 occurrences) was ‘Usability of technology’. The quotes in this code were more general, indicating that usability is a requirement for working with technology.

Role divisions and ways of working (119 quotes)

Participants also spoke about the context in which technology could be used. The codes in this theme could be divided into three groups: the therapist role, the client role, and the role of technology. First, for the therapist role, the most prevalent code was ‘Therapist dependent’ (64 occurrences). This code referred to elements of compassion that participants found more dependent on the therapist than on the platform. It was most often linked to the compassion element of understanding the universality of suffering (23 times): “The platform helps me not to judge a client’s problems’, no, I think that says something about how I think about it, and I don’t think that is necessary for a platform” (pp9). Almost as often, this code was linked to distress tolerance (21 times). Within the therapist role, the next code was ‘Colleagues’ (7 occurrences), which was always linked to the element of distress tolerance. This code shows that colleagues are important in dealing with difficult feelings. The final code within the therapist role was ‘Role unclarity’ (2 occurrences), illustrating that technology can blur the role of the therapist: “In suicide prevention, with self-registration, we also had to consider: should practitioners have direct insight into this or not? So that as a practitioner you are obliged, actually, to monitor what the client does. So that could mean that you overload the practitioner…” (pp23).

Next, for the role of the client, the most prevalent code was ‘More active role’ (11 occurrences), with quotes illustrating how technology can give the client more autonomy and self-management. Another code about the role of the client was ‘Relevant for clients, not professionals’ (9 occurrences). This code was mostly linked to the element of understanding the universality of suffering (5 times): “A platform made to treat people in mental healthcare should not primarily help me realize that anyone can suffer, but primarily be there for the client and should support me in this” (pp23). Next was the code ‘Meeting client’s preferences’ (7 occurrences), illustrating the importance and benefits of personalizing the digital or blended treatment. Finally, there was the code ‘Less active role’ (2 occurrences), describing how technology could also give clients a more dependent role.

Furthermore, participants made comments on the role of the platform. In the code ‘Integration of technology in the workflow’ (8 occurrences), participants spoke about their ways of working with technology. In most quotes, participants working for the completely online mental health foundation gave examples of working with technology in a structured manner. Another code was ‘Platform dependent’ (7 occurrences), about how participants found the importance of elements of compassion to be dependent on the type of technology: “If the medium is chat, then I am the instrument to alleviate [suffering]. While if the medium is a module, then I assume that the platform will partly do that for me” (pp9).

Research question 3: final adapted scale

We used the findings from the previously discussed research questions and the feedback from participants to construct the final scale. First, we selected the items for each element of compassion that were sorted as most important by each cluster. Next, we added an instruction to the scale, since many participants interpreted the items as implying that a digital mental health intervention was necessary for compassion to occur. The opening statement explains that the items refer to whether an intervention contributes to various treatment aspects, not whether it is absolutely necessary for them. Then, we phrased all items in the same way, as a continuation of “The [DMHI] helps me to…”, to keep all items similar and make the scale shorter and easier to fill in.

Furthermore, some of the selected items were adapted based on feedback from participants (see Table 5). To decide whether to incorporate feedback, we considered whether adapting the item could improve it, for example by making it clearer or more concise, while remaining in line with the theoretical conceptualization of compassion by Strauss et al. [1]. We did not add any additional items that were suggested by participants, as these did not directly relate to compassion (e.g. items around evaluating differences between automated and non-automated responses). For a full description of all feedback and additional item ideas from participants, see Additional File 4. The full version of the final scale can be found in Table 6, while the short form of the scale can be found in Table 7. Both versions of the scale can be used with a 5-point Likert scale (strongly disagree = 1, disagree = 2, neutral = 3, agree = 4, strongly agree = 5). Additionally, a randomized and ready-to-use version of the full CTS-P can be found in Additional File 5.

Table 5 Adaptations made to selected items with reasoning. DMHI = Digital Mental Health Intervention

Table 6 Full version of the Compassionate Technology Scale for Professionals (CTS-P) with non-randomized items

Table 7 Short form of the Compassionate Technology Scale for Professionals (CTS-P)

Discussion

The main goal of this study was to develop a new scale for mental health professionals to evaluate Digital Mental Health Interventions (DMHIs) on the extent to which they facilitate compassion between them and their clients. We did so by involving mental health professionals themselves to explore relevant items for such a scale. Based on our findings, we developed a full version and a short form the Compassionate Technology Scale for Professionals (CTS-P).

Main findings

First, we identified three clusters of professionals who prioritized different items. The first cluster prioritized goal-oriented items, related to the compassion element ‘acting to alleviate suffering’, and found the realization of the universality of suffering least important. This cluster overlapped with the second cluster, but the latter focused more on the importance of recognizing suffering and empathizing with the client, and least valued the support of a DMHI in dealing with their own difficult feelings. The third and smallest cluster differentiated itself more from the first two clusters. Here, participants found it important that a DMHI assists in accepting one’s own difficult feelings as a therapist. Interestingly, this cluster also highly valued a DMHI facilitating affective empathy, while items related to cognitive empathy were sorted as least important. A possible explanation for the differences in prioritized items by each cluster could be a potential difference in therapeutic backgrounds. For example, a more goal-oriented approach could be linked to Cognitive Behavioral Therapy, a focus on empathizing linked to more Client-centered therapies, and highly valuing the acceptance of difficult emotions in the client and in oneself could be linked to Acceptance and Commitment therapy [46]. However, therapeutic backgrounds were not recorded in this study, and generally did not come up spontaneously during the task or the interview.

Overall, the majority of participants valued technology’s contribution to recognizing suffering, empathizing, and alleviating suffering the most. These three elements show similarities with the Sense-Think/Feel-Act paradigm, which is considered a broad roadmap in the field of human–computer/robot interaction [47]. In this paradigm, ‘sense’ could be related to recognizing suffering, ‘think/feel’ could be related to cognitive and affective empathy, and ‘act’ could be related to acting to alleviate suffering. Participants valued the remaining elements of compassion less in relation to technology: the awareness of the universality of suffering and distress tolerance.

Furthermore, participants shared their associations with items on technology and compassion. These could be divided into three main categories; benefits and possibilities, critical or nuancing notes, and role divisions and ways of working. The main envisioned or experienced benefit of technology for compassion was to obtain more information about the client or to be closer to the client. This benefit could help facilitate recognizing suffering and empathizing with the client, according to participants. This finding is similar to one of the benefits found in an earlier interview study with psychologists [29], where participants stated that regular contact through DMHIs makes the therapeutic relationship more intimate and personal. This finding contradicts the perception of many healthcare professionals that technology can detract from compassionate care [11]. Moreover, a higher contact frequency in treatments has been shown to lead to significantly faster recovery from psychological distress [48]. Other mentioned benefits were that technology can support the professional in coping with distress or reduce stigma in mental healthcare, in line with the concept of ‘beneficent dehumanization of care’ which was described in earlier literature as contributing to compassionate care [49].

The large majority of the critical or nuancing remarks was about how a DMHI was not necessary to provide compassion in treatments. As mentioned earlier, the majority of participants found DMHIs particularly unnecessary to be aware of the universality of suffering or to help them accept their own difficult feelings that can arise in treatments. Especially for these elements, it seems that compassion is a basic prerequisite for any mental health professional, for which they should not need technology. Compassion is seen as part of the expertise and motivation of mental health professionals [50, 51]. Without clear ways of integrating technology in the workflow, or a clear framework of compassionate technology as a synergistic collaboration with roles for both humans and technology, it seems to be experienced to meddle in or endanger this expert-role [21, 52, 53]. Therefore, we added an instruction to the scale, stating that the items were not intended to refer to DMHIs as the only way to offer compassion in treatment, but rather as a supportive addition. Ideally, the integration of DMHIs could lead to increased empowerment and self-management for clients, and new therapeutic roles to optimize treatments [54].

The compassionate technology scale

To reach our main aim and develop the Compassionate Technology Scale for Professionals (CTS-P), we selected the items that were prioritized by each of the three clusters from the Q-sort, but also made sure that each element of compassion was represented equally. In this way, we balanced the input of participants with a proposed definition of compassion from scientific literature [1]. Although none of the clusters found it important for DMHIs to support awareness of the universality of suffering, this element was still included in the scale. We had several reasons for this choice: first, the five-element factor structure of compassion has been empirically supported [55]. Second, being aware of the universality of suffering distinguishes compassion from related but different values such as pity [1]. Third, Sinclair et al. state that separate elements of compassion on their own are not inherently compassionate, and that it is their combination that forms compassion [7].

However, this does not mean that every DMHI needs to facilitate every element of compassion in a blended treatment. As recommended by Morrow et al., compassion can be viewed as a motivational caring system, in which both humans and technology contribute to compassion [21]. Taking on a motivational caring system approach to compassionate technology could also imply that depending on the target group and context, certain elements of compassion are more relevant for respectively a DMHI or mental health professional to support than others. The CTS-P could provide insight into which compassion elements a DMHI facilitates and which it does not, fostering reflection on how the mental health professional and technology can complement each other.

This is the first scale to evaluate DMHI use on compassion. Existing related scales have focused on the evaluation of the therapeutic alliance with a virtual therapist [56] and adapting existing therapeutic alliance scales to the use of a mobile phone app [57,58,59], a conversational agent [60], virtual or augmented reality [61], and guided internet interventions [62], but not on evaluating compassion. Studies show that mental health professionals experience pressure from managers and insurance companies to use DMHIs for economic reasons, which can lead to distrust [29]. Compassion offers an important contrasting motivation for DMHI use that suits mental healthcare more: matching the values and needs of professionals and clients, instead of using technology as a quick fix [50]. The Compassionate Technology Scale can help evaluate whether this value is present in working with technology, and if so, if this leads to more meaningful integration of DMHIs in treatment.

Strengths and limitations

In the development of the Compassionate Technology Scale, Q-methodology proved to be a helpful approach for involving stakeholders. Our data showed which items participants prioritized, helped to represent different perspectives among professionals, and gave space for feedback and insights into how participants interpreted items. Thus, Q-methodology fits well with recommendations to involve stakeholders early on in new eHealth developments [63]. Moreover, participants enjoyed being able to share their reasoning and experiences, although the task was also experienced as cognitively challenging.

Regarding limitations, the large majority of the participants in this study was female (93% and 87% respectively for the quantitative and qualitative data). This female majority is also found in the field of healthcare at large, with 81% of people working in this field being female [64]. However, since some research shows gender-differences in compassion [65, 66], this lack of gender diversity could have influenced our findings.

Finally, a potential limitation of this study is the relatively small group of participants. However, in Q-methodology, the diversity of participants is more important than the number of participants. The rationale behind the participant count is based on Q-methodology's focus on identifying distinct perspectives rather than quantifying their frequency within the population [36]. The essential criterion for sample size is the adequacy for discerning stable factors, with 20–50 participants usually being sufficient [36]. With 15 participants this study successfully identified three distinguishable and interpretable viewpoints, indicating that the sample size was adequate for achieving saturation in our analysis and meeting the study's objectives.

Future research

First, further research could evaluate the CTS-P with a different sample of mental health professionals as a cross-validation. Next, future research with a psychometric focus is important to corroborate the factor structure, validity, and reliability of the CTS-P. Furthermore, future studies could employ the scale to explore which compassion elements are more and less important for a DMHI to facilitate in different target groups and situations, and whether the importance of compassion differs between types of interventions. As such, the scale could contribute to further shaping compassionate blended treatment. Now that this scale has been developed for the perspective of mental health professionals, another logical next step is the development of a client version of the scale, which is already in progress. Here, items are reframed to focus on the client’s evaluation of compassion when a DMHI is used.

For future studies using repeated measures and experience sampling designs [67], we developed a first short form of the CTS-P consisting of three items. For many DMHIs, short daily interactions are common for both the client and the therapist, and studying the dynamics of possible contributions of this technology to compassionate care would be supported by this short form [68]. Currently, the items in the short form are overarching, in the sense that they contain the term ‘compassion’ and therefore allow for a more general assessment of the compassionate technology construct. Further psychometric studies of the full scale could indicate which item is most representative for each compassion element, leading to an alternative short form with five items that do not contain the term compassion, but instead represent the five elements of compassion also found in the long form of the scale.

Finally, in their systematic scoping review, Morrow et al. made a theoretical link between compassion and artificial intelligence (AI) technologies [21]. They reconceptualized compassion as a human-AI system of intelligent caring, adding an additional element: feedback on whether suffering was actually alleviated. On the one hand, future research could explore whether this sixth element should be taken into consideration when conceptualizing compassion or evaluating DMHIs in terms of their contribution to compassionate treatments. On the other hand, this proposed sixth element of compassion could also be seen as an outcome of compassion, while the other five elements are part of the process of compassion. In that case, mediation or moderation analyses could explore whether technology contributes to reaching this outcome of having alleviated suffering.

Conclusions

Through a combination of qualitative and quantitative research, participant input and a theoretical basis, we developed this first version of the Compassionate Technology Scale for Professionals. This scale can be used by mental healthcare providers to evaluate whether DMHIs contribute to compassion in treatments. Moreover, by developing this scale, we aim to facilitate a movement towards a new motivation to design and use technology for mental healthcare, starting from compassion. Our scale can make compassion in these situations visible and comparable, and thus make it possible to consciously steer towards more compassionate ways of working with technology in mental healthcare.