What the textbooks don’t teach about the reality of running a digitally enabled health study: a phenomenological interview study

Abstract

Background

Most studies do not produce their intended outcomes on time or within budget. However, it is challenging to identify the facilitators and barriers to successful study management when the “behind the scenes action” of especially digitally enabled health research studies are akin to a black box. Therefore, it is necessary to explore first-hand experience of the facilitators and barriers to managing digitally enabled health studies. The goal of such studies is to produce new knowledge and/or develop tools that can be translated to real-world benefits for the health and care sector, individuals, and other stakeholders. These studies now exist in a time that encourages collaborative research activities with interdisciplinary research partnerships, industry collaboration, end-user involvement and insights for policy. These expectations require teams with different work cultures, methodologies, technologies, and approaches to work together, resulting in significant benefits but also challenges.

Objectives

To explore the relationship between the dynamics and needs of research teams and the technology used to manage digitally enabled studies through the experience of those who worked on such studies.

Methods

We used an interpretive phenomenological approach to explore research team members’ experiences and perceptions of study management in the field of digitally enabled health research. We interviewed 15 research team members from eight studies. A semi-structured interview guide was used to explore concepts related to study activity management, team dynamics, resources and technologies used to manage research activities, and reflections of personal experiences. An adductive thematic analysis was performed on the transcripts.

Results

Five main themes were identified: 1) Project Team, 2) Study management, which included management technologies, 3) Study plan, 4) Intervention, 5) Participants. This paper focuses on the first two main themes. Sub-themes included: Roles and responsibilities, Methods, Changes, Challenges and solutions and Expectations vs. reality. Sub-themes were applicable to all main themes. Therefore, results were presented as knowledge gained from the interaction between sub-themes within each theme, i.e. referred to as “comprehensive insights” in the results section of this paper.

Conclusion

This interview study provides new knowledge about the realities of working in collaborative, digitally enabled health research studies and demonstrates several opportunities for improved understanding of study management. More realistic and thorough understanding of the complex system in which digitally enabled health research exists can be applied to better prepare experienced researchers and newly graduated students entering the field, as well as improve existing strategies for management.

Highlights

1. Research team members with developed skills from other fields can add value to research projects by contributing relevant yet different perspectives and solutions, regardless of a lack of experience in digitally enabled health studies.

2. Working together while having different approaches to science, e.g. building knowledge subjectively via experience (constructivism) vs. gaining knowledge objectively via observation and structured study (positivism), requires us all to understand the purpose of each approach and openly acknowledge one another’s contributions.

3. Researchers are eternal learners – given the speed of technology development and work cultures of collaborating non-scientific research-based partners, we need to expand our knowledge of methodologies and research approaches to keep pace and relevance.

4. The use of study management technologies was largely unplanned. Research team members chose and used systems and programs that they were familiar with and available at the time.

5. The multitude of study management technologies mentioned were individually focused on a study stage or task, largely siloed, and subsequently challenging for collaborative tasks.

Introduction

There is a lack of transparency in research reporting about the resources, time and funds used for health research. What little we do know about the current way of performing clinical trials, such as pharmaceutical studies, is that almost 80% are delayed, and more than two-thirds do not reach participant recruitment goals [1]. However, digitally enabled health studies have the resources and potential to overcome some of the traditional barriers that drug and bio-medical studies face. In their article about how to improve clinical trials, Michael Christel highlights the potential of digital health to implement “process improvements” such as digital recruitment strategies, remote and virtual data gathering, or maintaining project team engagement and participant support services [2]. Other literature focuses on evaluating the objective factors for success and failure of a project based on the traditional outcomes of time, cost, and productivity [3]. Very few include subjective factors as project team member experiences. However, these are mainly evaluations of information communication technology (ICT) for use in the healthcare sector, not in research project management [4]. Therefore, there is limited evidence about what goes on behind the scenes of real-world studies. This makes it difficult to determine how to make digital and health research faster, more effective, safer, and more relevant for patients and other end-users. So, what affects the progress and outcomes of a digital health intervention project? Is it the resources or technologies we use? Or, how we use them? Our professional skills? This is what we aimed to explore in the presented study.

Layers of digital health intervention research

Digitally enabled health interventions

In this paper, we include in the definition of digitally enabled health interventions both interventions that use digital tools to administer or manage the study and/or digital tools as part of the intervention itself. Digitally enabled health interventions exist at the crossroads of several factors: personal, psychological, physical health, and to change, the healthcare system, the healthcare authorities, the development and supply of technology, the needs of users, and more.

Collaborative research activities

The field of digitally enabled health research is unique in the type of technologies used, the intersectionality required and the speed at which results are expected. Collaborative research activities are those in which distinct groups, e.g. academic fields, industry, members of the public or end-users, work together to generate knowledge about a solution that can then be applied to the real-world [5]. The dawn of digital health has forced the areas of industry and academic research to work together to produce practical technology solutions for individuals and the health and care sector.

Whereas these organizations were once fundamentally separated by oceans of epistemological differences—characterized by different financial agendas, work cultures as well as production speeds, outcomes, and methodologies, among others [6]- they are now called to work together in these collaborative research networks [7, 8]. “Joint undertaking” and “public–private partnership” are buzz words in grant calls. Funding schemes such as those by Horizon Europe, EU4Health and Digital Europe strongly encourage such partnerships for funding [9].

For the purpose of describing the complex context in which such collaboration must function, it is important to note that health innovation research projects also call for the participation of those who will eventually use the technologies [10]. The concept of “participatory design” has become not only called for by health authorities but also necessary to maintain relevance in today’s world [11, 12]. As a result, we, as digital health researchers, are in a world where we have to [5]:

• 1) Produce knowledge of the impact of these digital health tools,

• 2) Address the interdisciplinary questions and needs that are relevant to a person’s use of technology for their health, which means,

• 3) Collaborating with interdisciplinary groups, industry or individuals who approach the project in a fundamentally different way, who may be located in a different geographical location (or with a different organizational or societal culture), while

• 4) Meeting the new health research standard of participatory methods (i.e., involving the public), who might also be remotely situated, and

• 5) Keep pace with the development of the tools in industry/world.

Collaborative project management technologies for digitally enabled health interventions

The adaptation of research project management to digital health interventions has become more complex. The uniquely uncertain and flexible nature of digital health interventions in addition to virtual work environments, require researchers and management technologies to be more adaptable. Research teams now need management tools that address researcher-to-researcher or partner and research-to-participant coordination during each stage of a study plan.

Technology can facilitate the smooth transfer of data and knowledge [13]. Further, communication has been found to be a primary barrier to the success of some collaborative research projects [14]. The connection between technology and productivity or success of a project is filtered through team member engagement. In the corporate or business world, there are centralized ICT systems—also referred to as project management information systems (PMIS)—in place that all team members are expected to learn and use [15]. Examples include software such as Microsoft Project or Oracle Primavera programs or, for smaller projects, EasyProject, ProjectLibre, or monday.com [16, 17]. However, in academic health research, projects are a collection of separate and established research teams who have their own distinct work styles and communication technologies, not to mention different perspectives and approaches to research and development of any new health technology [18]. Project management and communication technology must balance the importance of cohesive and clear communication with meeting the work style needs of its very different users.

There are relatively new frameworks for how partnerships should be modelled, these give primarily broad strokes, leaving both sides of the partnership to interpret these recommendations as they see fit. However, even the most instructive frameworks, e.g. the framework for university-industry collaboration described by Awasthy et al., lack details of how project management plays out realistically during a study or what tools can be used to facilitate the tasks of managing a project [8, 19]. There are also organizations that have developed ICT systems specifically to facilitate academic-industry partner collaboration [20]. However, current products and services are only available at a cost, raising questions about feasibility and awareness of such systems among researchers. A review of collaborative research projects finds that the current knowledge of such collaborations still has some significant gaps, with digital platforms being underutilized [19].

In this paper, we aim to explore the relationship between the dynamics and needs of research teams and the technology used to manage studies. Our research questions were:

• RQ1. How are research team members coping with the dynamic and complex nature of digitally enabled health intervention research?

• RQ2. What technologies and strategies are they using to manage their digitally enabled health research projects?

Methods

Study design

This interview study was part of a larger project, “New methods for evaluation of digital health services, eHealth and mHealth – a study of a dynamic concept for effective studies”, which, in addition to exploring study management strategies and factors, evaluated an online study management system [21, 22]. The core project included interviews as a sub-study to learn from the experiences of others in order to improve the functionality and future implementation of the developed system in health research [21].

Methodological approach

We used a hermeneutical or interpretive phenomenological approach to explore research team members’ experiences and perceptions of study management in the field of digitally enabled health research. This phenomenological approach frames the description of a situation, a phenomenon, as a reflection of the person giving the description, their own perceptions, roles and feelings as well as the words used in their description, within the objective context of the phenomenon [23]. Our research peers’ descriptions of study or administrative factors that we, as researchers within the same field, may take for granted, could be the answer to understanding why and how certain studies are more “successful” than others. Generating a more realistic understanding of how digitally enabled health studies are performed is important for not only practicing researchers but also budding young researchers entering the field.

The presented study has a unique feature in that those being interviewed share similar work experiences as those performing the interviews, i.e. both groups have experience in performing digitally enabled health studies. This position provides a valuable insight into the lived experiences of our informants, e.g. what matters when performing such studies and what is meant by “success”, while still acknowledging that we can learn from our peers’ unique experiences. As such, each interview discussion was unique and tailored to what was important to the informant about their experiences and treated as more of a dialogue or co-creation of the conversation between peers, with the interviewer focusing on listening and requesting further explanation or clarification of a statement [24].

Interview guide

The semi-structured interview guide was based upon a review of both scientific literature and those from other fields, e.g. industry, business, etc., related to project management strategies and factors that affected their outcomes. Some of the core topics explored included the following: the published factors of “success” and “failure” of a study, impact of leadership and work-place cultures, virtual teams (as these interviews were performed shortly after COVID lock-downs were lifted), trends of project management knowledge and specific reasons why health or scientific research studies do not finish on time or within budget [25,26,27,28,29,30]. The intention was to apply these concepts to the interview guide to explore their application in the context of digitally enabled health study management. The interview guide was iteratively developed between three researchers with backgrounds in informatics (EÅ), health research (MB) and social economics (EB). We considered “success” to mean completing a study within budget and proposed timeline, as well as producing knowledge that contributed to the field of study – regardless of whether the original protocol was completed, or positive results obtained.

This was originally a mixed-methods study; the semi-structured interview guide, which focused more on the human experience, was meant to accompany an “inventory” of more objective and quantitative measures of study performance, resources and technology used for each stage. However, this was deemed too long and time-consuming for most research team members to answer, so it was removed. Interview questions covered the following topics: a) personal experience, engagement and intentions, b) communication and planning, c) inter-personal relations/interactions, d) general and other thoughts as well as any resources or technologies used in these situations (added at the end of each section, not formally included in the interview guide). A copy of the interview guide can be found in Additional File 1.

Sample

Informants were identified from a list of authors from our own previous scoping literature review of methods and measures used in digital health intervention studies [31]. A total of 63 authors were identified from a list of 31 studies that evaluated single mobile health (mHealth) apps and/or mHealth systems involving more than one tool between 2015 and 2019. There were no limitations based on the country where the study was performed. We aimed to recruit two research team members from each study.

Recruitment and data collection

Author informants were invited via e-mail to participate in a 1-h one-on-one interview. The invitation e-mail included a brief description of the study and a formal invitation letter with more details. The letter described their rights as participants and general privacy measures to ensure a safe environment for their honest feedback and contact information for the first author (MB). Their response was taken as informed consent. The invitation letter can be found in Additional File 2.

As this study took place during the COVID pandemic and many research team members were in different cities, digital interviews were performed primarily using Zoom and Microsoft Teams. All interviews were audio recorded and transcribed. Interviews were held in English by MB, except for two interviews that were held in Norwegian by EB and a research assistant. MB has several years of experience in qualitative data collection methods, such as focus group meetings and interviews, as well as the topic of digitally enabled health studies. Remote one-on-one interviews were chosen over group discussions primarily to allow interviewees to express thoughts they may not otherwise feel comfortable sharing with peers due to the threat of judgement. At the beginning of each interview, informants were asked for their consent to audio record the interview. The interviewers (MB or EB) repeated that any identifying information about informants, their colleagues, partners, the study or intervention would be removed during transcription and analysis and encouraged honest feedback. Informants were told they could contact the interviewers at any time before anonymization to withdraw their interview from the study. Recording only started after oral consent was given by the informants. Audio recordings were transcribed verbatim and stored securely. After each interview, the interviewers (MB or EB) asked if other team members from the study held a different position than themselves and whom the interviewers (MB or EB) could contact. Interviews were conducted between March 2020 and December 2022.

Reflexivity

Researchers involved in this study represented a variety of fields and approaches to research, including informatics (EÅ, MM), economics (EB), health sciences and business administration (MB), as well as those who specialized in psychology (EG) and public health (DL). Several team members were part of developing an online study management system as part of a previous larger project (EÅ, MM, and MB). During the study period, EÅ, DL and MM pursued the creation of a company around this online study management system. MB, who conducted most of the interviews, notes that their own experiences in digital health and interdisciplinary research allowed them to express a personal understanding of the situation to the informants, reflecting an intersubjectivity, genuine interest, and a more natural discussion.

Analysis

Both inductive and deductive approaches, i.e. adductive approach, were used to analyze the anonymized transcripts primarily by two researchers (MB and EG). Inductive thematic analysis, as outlined by Braun and Clarke [32], aimed to answer RQ1 by identifying research team members' experiences and perceptions of the study and team. Codes were grouped into higher level codes, i.e. sub-themes and eventually themes. A more deductive approach was taken as the last stage to note which of the themes reflected the major interview guide topics: a) personal experience, engagement and intentions, b) communication and planning, c) inter-personal relations/interactions, d) general and other thoughts as well as any resources or technologies used. Deductive thematic analysis was used after the inductive analysis to answer RQ2 by identifying which technologies and resources were used during each study, by whom and how. The choice of abductive analysis was based upon the intention to allow informants to share what they believed was important whilst ensuring that they described experiences throughout all the study stages (i.e. RQ1). This information was then used to contextualize how and which technologies and other tools were used during those experiences (i.e. RQ2), which were more explicitly asked for during the interviews. Questions about whether the strategies (RQ1) and resources (RQ2) facilitated or challenged informants’ studies aided in the interpretation of the information in a more subjective manner. Comparing responses from two informants from the same studies, i.e. triangulation analysis, allowed us to confirm or expand accounts of study activities and compare experiences [33].

Emerging codes and eventual themes were identified independently by the two researchers (MB and EG), who discussed iteratively amongst themselves and then presented findings to the research team. Other team members were less familiar with the details of the transcripts. The process whereby other team members reviewed the accuracy of the coding compared to the data of the transcripts contributed to the “defamiliarization” stage of abductive analysis, which encouraged a reconsideration of assumptions by the two primary analyses (MB and EG) and more robust interpretation of information [34]. Discussions continued until all agreed that we had reached saturation in terms of emerging themes, and it was decided that no additional interviews were required [35]. One researcher (MB) consolidated both researchers' results and additional input from the research team into commonly agreed-upon coding and thematic results. Analysis was performed manually in Excel and Word.

As described by Timmerman and Tavory’s article, abductive analysis and interpretation are influenced by the existing knowledge and “cultivated position” [36]. This process was iterative, and the foundation of our interview guide, i.e. interdisciplinary team management and work-related technology acceptance, formed the overall context in which we situated these findings. Due to its balance of diversity between gender, age and educational background, the positionality of each team member in this interview study provided a comprehensive and informed interpretation of the findings and presentation in this paper.

Results reporting follows the standard for reporting qualitative research (SRQR) guidelines [37].

Assessment of rigour

De Witt and Ploeg’s proposed a framework for the critical appraisal of rigour for this type of study – interpretive phenomenological interview study – which includes five concepts [38]. The first—balanced integration was demonstrated by the strategic recruitment and the breadth of interview topics from team members’ intentions to reflections about the performance of study management and what was important to them during the experience, which were based on both research into the background of study administration, current trends, and our own experiences in the field (see Introduction). The tenets of interpretive phenomenology were also evident throughout the methods and analysis, including focus on lived experiences to explore underlying reasons, decisions, and human factors, for study “success” or “failure” beyond the objective stages and requirements of a study (see Interview guide) [24]. The second – openness, was evident in the explanation of our – as the researchers behind this study—perspectives and positions in relation to the research topic (see Methodological approach). The third—concreteness was evident in the explanation of the background of the study and its context in the research and health realms (see Introduction) as well as the usefulness of this line of inquiry as lessons for us and our peers to learn from. The fourth – resonance, was reflected in the depth of inquiry and richness of the data. The fifth – actualization, is evident in the potential applicability of these findings to both veteran and budding researchers in this field (see How to apply lessons from our peers, below) [38].

Ethical approval

The interview study was found to be exempt from the Regional committees for medical and healthcare research ethics (REK) Nord approval (ref. 625,936). as part of the project “New methods for evaluation of digital health services, eHealth and mHealth – a study of a dynamic concept for effective studies” [22]. Ethical approval was then pursued and approved by the Personal Data Protection Officer (Personvernombudet) at the University Hospital of North Norway (UNN) (ref. 2023/8767, 03190).

Results

Participants

Of the 63 authors from the publication references and eight who were recommended by informants (some of whom were also in the reference list), 15 agreed to participate. These 15 researchers (n = 13 women) represented eight international studies involving eight countries. We successfully recruited two researchers from each study, at least one from a more managerial or senior researcher position, apart from one study for which only one informant could join. All interviews were one-on-one, except one in which both researchers decided to perform the interview together. In the results, we denote an informant by the number of their study (1–8) and either A or B, which only denoted which interview was performed first, not their position within the research team. Additional information about each informant is presented in Additional File 3.

Themes and paper structure

Five overarching themes were identified: 1) Project Team, 2) Study management, which included management technologies, 3) Study plan, 4) Intervention, 5) Participants. This paper will focus on the first two main themes. Sub-themes included: Roles and responsibilities, Methods, Changes, Challenges and solutions and Expectations vs. reality.

Figure 1 illustrates these two main themes, which research question they answer and the order of sub-sections within the Results. The sub-themes are not explicitly listed here because one sub-theme could apply to several main themes. It was, therefore, difficult to report them separately or in sub-headings. Instead, sub-themes are represented within “comprehensive insights” as it applied to the research questions (Fig. 1). See Additional File 4 for a description of how sub-themes are represented as “Insights”, which form the Results sub-sections.

Fig. 1

Illustration of Research questions, Themes, Insights (i.e. Results sub-sections), and their order within this text

Theme 1—Project team

Insight 1: Gaining experience on the job

Informants were asked several questions about their intention for the study, motivating factors about working on their project and any challenges that they experienced personally.

Previous experience gave informants a sense of control and comfort in the area of digitally enabled health studies [Informant 3A: “we had all done clinical trials before, so we knew the process of applying to the ethical committee…how to run it… in what order and how to have quality in our work”]. However, one-third of our informants were not experienced in digital health studies (Additional File 3). Some were not versed in the specific role they performed, while others were foreign to the research area [Informant 1A: “In a way it was…not at all my research area and of course that was also a bit difficult for me during the project…leading something that I did not know so much about”]. Project management was seen as a difficult task for many researchers [Informant 1A: “we are not educated to do [project management]”].

Not knowing a field or role was not a reason not to try [Informant 1A: “I am not a technology person. I am a nurse…I was in a way just put into this leading role, so, I think in the start I did not know very much about how I should do it…it was trying and failure”]. The researcher role was described as more of a continuous learning opportunity – an opportunity to be a perpetual student [Informant 8B: “I am not an expert on [statistics]. I have to read a lot of this and study it a lot of this. I still have some things that I don’t understand fully”].

Informants’ motives for joining an unfamiliar territory were the potential of technology in health and the opportunity to work with interesting colleagues [Informant 6B: “[I] learned about [PI’s] team and this new avenue of digital medicine…got really excited about it and decided to transition into that role where I could work on a really cool team and also be at the forefront of technology”].

Insight 2: Building the research team

When asked to describe their team, informants focused on the structure and intention behind the project teams they hired and worked with. Ensuring that the project team had the necessary competencies happened during hiring and recruitment [Informant 4A: “when bring new people in, I know what we need to train them in and where to put their skills. We will usually have a mix. They need good administrative skills for the project management, and then they need good research skills for different components”].

Most informants noted the importance of their team members [Informant 4A “I think it is the people who make [a project] successful”]. The diversity of team members’ backgrounds was considered a strength of interdisciplinary projects. Two common challenges to this ideal team-building situation were finding unique and necessary competencies in the relatively new field of digital health and time.

In more immediate situations, such as when the need for additional staff is emergent, time to perform such thorough vetting is not always available. In Study 6, the unexpected need for patient support services required hiring, training and receiving ethical approval for additional staff mid-study.

Insight 2.1: Impact of team members’ attitudes and support

Informants were asked about the interpersonal relationship within the team [Informant 4A: We had very different disciplinary lenses—we had very positivist thinkers and very constructivist thinkers…Still, being able to bridge that epistemological divide was really encouraging. I think it is because those people were just so invested and wonderful and wanted to learn.”].

Setting aside personal ego or professional agenda had a positive effect on the team’s ability to work toward a common goal. The concept of having “common ownership” meant considering the project as the team’s project and not only the principal investigator’s (PI).

Support from one’s team members could make up for the challenges of interdisciplinary work. Despite being confused about the project, support from team members kept some informants engaged.

Informant 7B: “I found that [the PI] was really receptive to my thoughts and ideas…that can be a frustrating experience when you feel your voice is not heard, but… [The PI] deferred to my judgement 90% of the time… and that just felt really nice…to have my voice and my expertise and my opinion and perspective be valued by the team. So that was probably what kept me going”

Insight 2.2: Team member engagement, not guaranteed

When asked if team members knew what was expected of them and when they were expected to do it, Informants described a sort of “team member attrition” over the course of a project. By the end of a project, most informants described that the responsibilities fall on the shoulders of a few individuals.

Reasons for decreased engagement were often competing responsibilities or being offered competing opportunities. One informant noted strong team member engagement at the beginning of Study 1. However, the other also noted the challenges of recruiting collaborators before the study.

Informant 1A: “Some were on sick leave... Other persons, they had new positions..[or] not very into it and were engaged in other things and were busy… engaged in teaching and other projects. So, it was not easy to get them to have time…both municipalities [for] long periods, it was difficult to get in touch with people at all”

Insight 2.3: Roles and responsibilities: the importance of one person, one role

As a follow-up question, we asked informants to elaborate on the realities of working with or managing a team in which many aspects of the project changed over time. In Study 3, dividing responsibilities amongst the team was seen as valuable and effective [Informant 3B: “[Technology manager] was amazing in this project, so it was extremely important to have a dedicated resource for the technical support. So [they] provided technical support to our patients here and also…to the local teams in the other countries”].

Study 8 was an iterative study, making it uniquely capable of learning what to change from previous iterations in order to optimize resources and workflow.

Informant 8A: “we rigged an organization around it in a different way. We had a project manager that was a professional project manager. We had me as kind of a daily manager…to do the logistics of the participants, to manage the staff…And we had one person who worked on HR, just recruiting new people and doing kind of the administrative work around that. And one financial adviser…Was a major job. And then we had one person to work, working on mediation. Recruitment of participants and outgoing information to the public”

While some argued for a dedicated person for major tasks, others realized this was not feasible [Informant 6B: “I quickly realized that I was getting overwhelmed. Trying to keep up with sending…100 devices per day…The project manager…had a lot of experience with running big clinical trials…I think she realized too that I was getting a bit overwhelmed].

Some informants noted their own challenges of holding dual roles [Informant 3B: “The optimal situation would have been to have…a project coordinator who doesn’t have the scientific responsibility,… There are a huge amount of activities which maybe one tends to underestimate when you start”]. Many faced the same barrier to hiring such a diverse research team—a limited budget.

Insight 3: The work environment

Informants were asked to expand upon what contributed to or hindered effective teamwork. Again, the concept of putting the project’s needs first was beneficial.

Informant 8A: “The environment that you work in is very… an essential thing. You have to have trust between the technicians [research staff]…the feeling of one pulling in the same direction, is important. Instead of “this is me and mine” and “I want to leave early” and “I don’t care if this one has to work two hours more because I leave”. So, you expect a lot from the technicians that work there. Then you have to lay a very good foundation”

Work environments were benefited by the project leaders’ acceptance of the reality of the research situation and value of research team members.

Informant 6B: “I think it really helped, [saying] “…you can all talk in our team meeting…We are all kind of learning together…You can offer some really valuable insights coming from an administrative point of view… that would be beneficial for the study,” I was letting people know that their talents were really valuable…I think was important in terms of setting the stage and making sure everyone feels comfortable in raising issues”

Creating a positive work environment was also challenging for others and a trial-and-error process that was developed over time. [Informant 7B: “hearing them out, but politely saying “No, we are not doing that right now,” or “…maybe we can incorporate that into the next phase,”…I started to really listen…I found my little ways to kind of incorporate their views…that helped to relieve a lot of the tension as well”].

Insight 4: Open communication and trust

Informants were asked to describe the overall communication and efficacy of their collaboration during main project activities. Communication and trust contributed to a healthy work environment [Informant 8A: “a safe environment for the technicians to bring up anything with any of us…[the project manager] was very good at…accepting whatever they had to bring in, listening to them and doing what she could about whatever they brought up”].

A common consequence of a lack of communication and common understanding was confusion and frustration.

Informant 7B: “[I] was kind of given a very “loosey goosy”…review what the project was about, but to be honest I had a really hard time at the time…fully understanding exactly what they were trying to do…There is still a part of me that still doesn’t even know what the whole full objective was…in my opinion it was kind of a very dysfunctional team…. the full proposal…was not shared…I have tried asking for it a few times…[the PI] seemed to hold that proposal…very closely”

A lack of clear communication resulted in inconsistent engagement by external team members during meetings. Subsequently, team members lacked understanding of the purpose of study tasks and provided unconstructive feedback. [Informant 7B: “I do think that the entire team sensed the lack of direction, and it was highly transparent in all of our team meetings.”].

In Study 6, people were encouraged to be honest about their work capacity so that tasks could be reallocated and workload shared. However, asking for help was not natural for everyone [Informant 6B: “I think maybe too often… being able to ask for help was really difficult… I said “I’m fine, I am just really beat”… I was not fine. I was so stressed… it was not normal to go at the pace that I was going…[if additional help was not hired] I would probably have a nervous breakdown”]. One reason for this challenge was a sense of duty, perception of expectation and responsibility to the project. Understanding work capacity was described as a process that required the support of others.

Insight 4.1: But we don’t even speak the same language!

Different “languages” can mean different work cultures, vernaculars or spoken languages. Informants were next asked about specific challenges experienced during collaboration.

Informant 1B: “Because of the background that the work-package-leaders…they were mainly from nursing… [and] It is very important [to them] to have a baseline. You need to understand the current situation…While in information systems…we also need to design a solution…where the actual resources are required, and understanding the problem is an iterative process. Because as we design something, we test it and then we also learn about the problem again”

When research team members’ agendas did not align, the goal of meetings became “managing personalities” instead of study tasks.

Informant 7B: “They truly are like the [Country’s] experts. They would always speak up… they were really adamant about doing it through their lens. But, again, when you have like six people and they were all really adamant that their way was the best way…it was more just… managing the bold perspectives and personalities of the team. I think that´s what made it difficult”

Insight 4.2: Collaboration, challenges, and compromise

Informants were asked to exemplify how such interdisciplinary miscommunications were resolved. Conflicts were described during every stage of the study. Often these conflicts took time and several meetings to clearly communicate both parties’ perspectives of research.

Informant 6A: “There was only one conflict…with the sponsor team of scientists…They were very very used to their trial…[they] expressed their concern about how slowly we were enrolling…it seemed that they were missing the whole idea of this iterative [recruitment] process and learning as we go…We ended up having two meetings together. The first meeting went poorly because it was them expressing their concern and me getting angry because they didn’t understand. At the second meeting…we explained the whole vision for the trial…And kind of proving our ability to them”

Sometimes, it was not always possible to identify the root cause of miscommunications during the course of the project [Informant 6A: “the person who put his reputation on the line to sponsor it…was told by many people, including his boss, that there was no way the trial was going to work…that we were going to be stuck with an unsuccessful trial and be blamed for things”].

Lengthening the recruitment time was another tactic when a partner was uncompromising in their enrollment goals. However, budget was one challenge that could not be changed. Some informants had to remain flexible and use outside resources to compensate for unfinished tasks that resulted from such miscommunications.

Informant 1B: “We had some issues with one of the [development] companies…I think it was a fuzzy responsibility issue… Basically, the project stopped, or at least the development part of the project. And in addition, the company had not really communicated about the hours they reported working. So, all of a sudden, their hours were done… They did not even have any documentation to provide us on the development process…we needed this technology development, but then we did not have the money”

Theme 2—Study management technologies

Informants were asked what technologies they used to manage their projects, including team management, intervention design and participant coordination. Note that we expect that the management technologies mentioned during the interviews are not an exhaustive list; more were used but not included in informants’ overview of used tools. A summary of the technologies and how they were used are described in Table 1.

Table 1 Digital and analogue tools used for managing digitally enabled health studies

Insight 1: “I prefer my own technology, thank you.”

An individual’s work technology choice was less of a choice and more a matter of availability and budget.

Informant 6B: “It was a lot of trial and error in terms of what worked best and what made the most sense at the time …what software we had available and…not going to be super expensive…Because we did not budget for them in advance… and could not realistically go back to our sponsors and say that “We need XYZ thing and this is what it is going to cost in terms of money. Please help us!””

Researchers at universities reported that they were required to use specific platforms because their organization had a subscription. These were seen as not ideal but mandatory. [Informant 7A: “[Microsoft] Teams it is a nightmare, because I have five different [Microsoft] Teams accounts”].

Several informants noted challenges when merging communication processes between interdisciplinary teams [Informant 4A: “I can adjust my internal team to use new technology, but when you are working with an interdisciplinary team… asking them to shift their process is going to be a non-starter. If they work on e-mail, that is how they work”].

Using different technologies often led to siloed work procedures. However, when others tried to impose a more “commonly understandable” and “accessible” workflow on some informants, they still maintained their own workflow and worked separately. The mentality was that it only needed to make sense to them, not others.

Insight 2: Which is preferable – control or ease of data collection?

When asked to describe the process and resources used during data collection, informants described the wide range of possible data sources and necessity of compromise. Informants described how some resources, such as paper questionnaires, were simultaneously challenging, as they required more time, and beneficial, as they gave a better opportunity to clarify responses while the participant was still there. Informants believed that having digital questionnaires meant having less control over data collection.

The use of third-party partners’ own databases, such as insurance claims, allowed some studies to perform participant identification as part of recruitment much faster than if they did it “from scratch” [Informant 6A: “We had a pool of about a hundred thousand people we could reach out to who we knew were very eligible for the study”]. However, miscommunication can affect the project at any time, requiring informants to redirect their focus and delay research activities [Informant 6A: “we purposively were going to send 1000 [recruitment] e-mails per week and learn…It was not just the fact that [the sponsoring insurance partner] thought the enrolment was slow, but it seemed that [the sponsoring insurance partner] was missing the whole idea of this iterative process”].

Many informants were additionally affected by the General Data Protection Regulation (GDPR) requirements, resulting in unexpected delays when transferring data from external registries such as electronic health records or national health registries. Study 6 also experienced an unexpected challenge when, as required to ensure privacy, participants returned the intervention device to the manufacturer for data extraction and anonymization. The anonymized data was then forwarded to the research team. However, the research team discovered that if there was missing data when participants did not use the intervention device, they could do nothing about it by the time they received the device data.

Some even considered the privacy and security measures for accessing and extracting data unnecessarily cumbersome [Informant 8B: “It took more than 1,5 years to get the complete data”]. For example, in Study 8, accessing data from an external database with high privacy restrictions required the research team to apply for permission each time they wished to access the data, let alone export it. Electronically exporting the data required additional work and reliance on another researcher who was difficult to reach. The solution to extracting the data was to use the level of access that they did have permission for [Informant 8B: “I decided to just copy it by hand because I could literally see the code on the screen, so I could just copy everything”]. In addition, metadata – a sort of code list – was required to translate and interpret the raw data into useful information. Only then could they perform the required analysis.

Even strategic and careful planning was no guarantee that protocols would go as planned [Informant 3B: We started all this process very early…the application to the ethics committee and everything was there and done…But then [the ethical review committee] were overloaded…too many requests… very long processing time before we got an answer”].

Insight 3: Frequency and effectiveness of communication platforms

Informants were asked about different technologies used during day-to-day communication, and to comment on their effectiveness. E-mail was the most common form of day-to-day coordination, although the number of e-mails and managing information exchange via e-mail was challenging.

Sometimes, the amount of e-mail communication was too little. For example, it was not always successful at prompting efficient responses, especially from outside teams with whom informants had infrequent or indirect communication. It could take sending several e-mails to a partner before receiving a response with the necessary information.

Relying on e-mail to exchange information could result in missing details [Informant 4A: “using platforms like Freedcamp to allow for better information sharing, because the e-mailing got a little hairy internally, especially as we had turnover of coordinators and assistants and students…some of the data management got a little hairy as we had handovers”]. When it was important to have the most recent version of a document to which so many made edits, e-mail was not the most effective platform [Informant 4A: “There were a couple of authorships “snafu’s” where people lost a thread in an e-mail and wanted to be on something, and then you are two-thirds of the way through, and then you have to figure out a way to get them back on to it”].

Just because a technology is easy to use does not make it ideal. WhatsApp was used to communicate nonsensitive information regularly within research teams in two studies. While this type of platform offered ease of use and access to colleagues, it could be overwhelming when messages came at any time of the day or night. Instead, informants opted to “drop out” of some chat groups.

Some had used platforms that only allowed one person to access a collaborative document at once, which required additional energy and communication [Informant 6B; “trying to figure out the right version of documents in the software that we were using…PM-ing [private messaging] someone like, “Hey, I am using the tracking document now. Let me know when you are off””]. Informants from almost all studies mentioned using Google Drive, Google Docs and/or Google Meet. Google Drive was beneficial when team members were able to work simultaneously on collaborative documents; it was efficient and reduced the potential for miscommunications.

However, using these platforms to share and simultaneously update documents only works when the research team members use them [Informant 7B: “everybody had access to [them], but nobody, I don’t think had even logged into it”].

Insight 4: Needs for future management tools: feasible or a pipedream?

Informants were asked to reflect on what they would change if they were to do their studies again, given their experiences. Project management software, such as Basecamp, Freedcamp or Google Docs, was preferable for more formal information sharing, whereas Slack, WhatsApp or Discord were suggested for more continuous communication. It was also important to clarify which platforms would be used for which task [Informant 4A: “making sure you are clear on the process for communication… You need to have different processes for communicating with the investigators who are still part of the team, but you are not going to change the way that they operate on one of twenty different things they are working on”].

When hiring new personnel, whether because of turn-over or team expansion, it was especially complicated to train someone new on a technology or workflow that was used and designed by the previous employee. [Informant 6B “before it was just me…And then adding more people, we had to figure out a new tracking system…where everyone could see the status and updates of each participant in the study…[so] we could all be on the same page at the same time”].

Planning for which technology to use and for which stage was something that many valued, but few were able to implement. All informants mentioned, at one point or another during their interviews, that automatic data extraction would have reduced time and energy.

Informant 5A: “If the [blood test] data had been extracted automatically, it would have been brilliant…[and] online questionnaires that they could have done at home… [And] if we could have replaced some of the physical consultations with a video call, it might have resulted in spending less time reorganizing appointments”].

However, automatic data extraction is not always the best option, especially when interaction with the participants is highly valued and/or necessary for the intervention [Informant 5A: “I would have lost the opportunity to talk to them and get close relationships with the participants”].

Discussion

This interpretive phenomenological interview study intended to explore the roles of teams and technology in digitally enabled health research projects. Three main themes were identified, of which this paper reported the first two – 1) the Project Team and 2) Management technologies. Sub-themes of these elaborated the more emergent results, which allowed us to answer our research questions deductively, as well as report more nuanced feedback from the inductive analysis. These ranged from the individual’s engagement and collaboration amongst team members and partners to which technologies and tools were used and how well they worked.

The overall goal of this study was to explore what affects the progress and outcomes of a digital health intervention project. Is it the skills of the research team members? Is it the resources or technologies they use? Or, how they use them? Short answer: all of the above, tossed together in a messy salad—perhaps with some secret ingredient we have yet to identify.

RQ1. How are research team members coping with the dynamic and complex nature of digitally enabled health intervention research?

The insights, i.e. the information gathered from the combination of sub-themes, primarily answered the first research question. Informants’ interviews reflect the collaborative, interdisciplinary, complex, and technology-enabled evolution of health research – from singularly focused studies to collaborative research activities. Informant 4A succinctly commented, “It is the people who make [a project] successful”. Several studies across research disciplines and industry consistently discuss the role of personalities, behaviors, motivations, and preferences in relation to the ongoing productivity or outcomes of a project [39,40,41,42]. Sutton et al. describe key factors of a successful interdisciplinary team, including a shared vision and vocabulary, explicit expectations of team members’ roles and responsibilities and continuous communication [43]. While most literature offers theories for team or study management or strategies for “this” or “that”, our informants have described the reality of performing digitally enabled health studies.

Research groups form a sort of micro-society. There are norms, hierarchy, expectations, work culture and even language indicative of that group, regardless of the field of study. When you join forces with another group, especially from a different field, there is a mutual culture shock of sorts that requires time and patience to navigate. A 2011 study by Randall, Resick, and DeChurch described a team’s “adaptive capacity” on the individual and strategy levels. They describe that it is fueled in part by the team’s psychological collectivism, i.e. individual team members’ affinity toward team-based work and putting the project goals above their own [40]. Our informants reflected this concept in their claims that “common ownership” of the project and adaptability were factors in engagement, productivity, team morale and collaboration, demonstrating that these concepts also apply to interdisciplinary digital health research.

The ideal of “one person, one role” was reported as a core factor in the success of the study. However, most informants described budget, time, and limited human resources as the most common barrier to this ideal. In fact, some described pulling “double duty”. Sutton et al. refer to this as “project leadership”, which merges the organizational and scientific level management roles in translational research, i.e. the process of bringing new evidence-based knowledge into clinical or personal self-management practice [43, 44]. As a result, several of our informants discovered the importance of respecting their own personal capacity within their roles, responsibilities and engagement over time.

Each project team member is chosen for a reason, i.e. their competencies, and represents knowledge and productivity. However, competence is not solely based on what you have been explicitly taught but also on how you apply learned skills based on lived experience. One-third of our participants had no experience in either the scientific field or the specific role they were responsible for within the project. Instead, their interest in the field and applicable skills motivated them, and a willingness and openness to different approaches and learning contributed to their success in their role. The role of competencies is consistent with Hana and Lucie’s [45] description of staff turnover as knowledge loss, and Briel et al. [46] reported that high clinical staff turnover resulted in poor-quality clinical trials. Our informants described how employee turnover posed a challenge to assigned roles and responsibilities when knowledge was transferred to a new employee, who now must decipher the old employee’s work and processes. However, another informant commented that the influx of new staff increased the diversity of prospects and skills in the team.

This “team member attrition”, as one informant described, was often due to other work opportunities, competing responsibilities in another job or project, or limited budget. Regardless of your background and enthusiasm at the beginning of a study – engagement in the project often varies over time. Kozlowski and Bell referred to this as “team continuance and decline” in their chapter about organizational psychology [41]. The result was the need to change – either protocols or personnel. Change was also a feature of digitally enabled health interventions that each project team member had to accept – sometimes begrudgingly.

Our informants reported that support from the project team leaders and senior staff was often a greater determinant of engagement when frustrations or challenges arose; it meant the difference between giving up and continuing their engagement in the study. Similarly, other literature has cited the leader as the one who sets the tone for the work environment, morals and ethical activities in addition to the goals of the project [42]. Professional training in project management is employed by large research organizations and has been found to be especially important when managing knowledge in interdisciplinary groups [43]. However, as one of our informants clearly stated, most researchers are not trained in study management.

There is clearly a gap between the ideals of team and study management, described by most literature, and the experiences of our informants. Especially for the growing and increasingly complex field of digitally enabled health studies, it is important to focus on the composition of teams, the team members themselves and why and how they use different strategies and tools to accomplish the goals of their studies[39].

RQ2. What technologies and strategies are used to manage digitally enabled health research projects?

The second research question was answered by informants’ feedback about how, why and who used which project management technologies. The answer to “which technologies were used” revealed a variety of specific programs, etc. (see Table 1), yet the commonality was that they were siloed, or disconnected. Informants noted several reasons why such combinations of technologies were chosen: technology for a specific task was not planned for and was only chosen when needed. This results in iterative periods of trial and error, during the study, to find the right technology for a specific task. Convenience, cost and familiarity were the most common factors influencing choice. Informants also noted that they were strongly encouraged to use technology, e.g. Google Drive or Microsoft Teams, and had to comply, regardless of their personal dislike of the decision. The main reasons were that their organization had purchased a subscription and made its use a standard in the workplace or others in their field were using it, and they needed to maintain collaboration. This highlights that each individual work culture and methodology is unique, and therefore, why interdisciplinary work between disciplines and industries is a significant challenge.

Although some of our informants were aware of comprehensive technologies used for study management, such as Freedcamp or Customer Relationship Management programs, it was unclear why these technologies were not adopted before the study started. Relevant literature suggests non-adoption of technologies can be traced back to one’s perception of, for example, ease of use and usefulness as a key indicator of one’s intention to use technology (e.g. measured by Technology Acceptance Model (TAM) or Unified Theory of Acceptance and Use of Technology (UTAUT) [47]. A study of project-driven organizations, using TAM, reported that, for projects with greater complexity, project management systems with superior ease of use and robust functionalities increased system usage and improved performance of project managers and outcomes [48].

Regardless of how advanced or comprehensive the technology is for improving workflow or quality outcomes, if users are unwilling to adopt it, the technology does not reach its potential utilization. Challenges to using technology, such as e-mail or social media platforms (e.g. WhatsApp), included an overwhelming amount of information that was difficult to keep track of for. Post-COVID assessments of technology in the workplace highlighted a duality; while they provide remote collaboration, faster communication and productivity, constant messages, and the influx of information to which one is expected to react immediately, negatively affect end-users’ mental health [49]. Other literature reported that not meeting meet in person and only communicating virtually negatively impacted social connections between teams and led to more frequent miscommunications [50].

Skoumpopoulou et al. argued that introducing new workplace technologies hinges on the human aspect – openness to change, perception of engagement costs and the rationale for the intention to use—more than the promise of new functionalities [51]. They argued that the unique attitudes of academics and their work environments toward individuality, freedom and independence affected the use of technology. The complexity of tasks also contributed positively to adopting new technologies, as the benefits of an easier and faster workflow outweigh the personal costs of investing in learning and use [51]. Because most of our informants worked with international or inter-regional groups within and outside their organizations, the concept of work engagement and remote collaboration is significant when considering technology for study management.

Available study management approaches and tools seem to focus either on the performance of study tasks or on administration and management. One study introduced a project management tool, initially intended for industry, the PRojects IN Controlled Environments (PRINCE2) approach, into a pharmaceutical trial. The authors described that detailed planning, standard reporting, and other consistent communication between different partners, i.e. funders, clinicians, research managers and researchers were reasons for successful adoption [52]. However, these results did not cover the study-specific and participant-facing tasks, and it was applied to a study with significantly more funding and rigid structure than those described by our informants. Another study that surveyed the adoption of technologies in different industries, including scientific research, focused on the study tasks that used information technologies (IT) [53]. The results are consistent with our informants’ use of clinical data repositories for data collection and storage, online ethical approval processes, and infrastructure. The diversity of technologies used for each major task in the cited study was also consistent with our informants’ descriptions of siloed technologies.

Collaborative research activities such as digitally enabled health studies are complex. Researchers are now required to include teams outside their disciplines or industries to explore, develop, and test interventions at the same speed as the production of digital health technologies and engage end-users throughout the process. There is a clear need to develop technologies that facilitate the unique needs of collaborative research project teams. The framework for evaluating such systems developed by Bani Ali et al. can help to determine the acceptance and impact of project management systems. Their proposed model and framework balance the impact between technology features, team member experience and needs and characteristics of the organization and purpose of project [48]. However, this model also confirms the feedback from our informants that these technologies may face resistance in the early phase because, as one informant described, most people prefer the technologies they have become accustomed to.

Strengths and limitations

The most significant strength of this study was diversity – both of an interdisciplinary research team who collaborated on this interview study and contributed to analysis, as well as diversity of informants from digitally enabled health studies, representing analysts, project managers, senior researchers, students, and research assistants from backgrounds within and outside of digital health. Strength also lies in the informants' honest and engaged participation, which provided rich insights into this topic. Two research team members were recruited from each study (except for one), which was used to triangulate experiences and develop a more comprehensive picture of each study.

A limitation of this study was the inability to collect more objective information on the resources and timeline of a study, as was intended for the mixed-methods approach. Because a survey of technologies, timelines and budgets was deemed too long and infeasible for busy researchers to complete, we could not collect an exhaustive list of these details. Instead, we relied on asking questions generated during the interview about any technologies used for a specific stage or task. Therefore, the information about resources, technologies and timelines may be incomplete. While the number of informants may seem low, the authors deliberately sent more e-mails than the number of expected responses to compensate for a potentially high non-response rate.

Furthermore, during and after the COVID pandemic, most people experienced technology fatigue due to video conferencing and reliance on remote communications [54]. We also note that the recruited informants represented 8 of 31 identified studies. However, the informants represented a diversity of experiences and roles that allowed us to reach data saturation for our main research questions, as no new themes emerged. Therefore, the number of informants was appropriate for the purposes of this interpretive phenomenological interview study.

How to apply lessons from our peers

Based on the informants’ experiences and perceptions, in addition to the authors’ similar experiences with limited time, budget and relevant human resources during digital health studies, we suggest incorporating more business or management strategies into research education, such as project and team management training, interdisciplinary skills development, and utilization of integrated project management systems. Effective interdisciplinary collaboration and role clarity are essential for implementing digitally enabled health research, due to the multitude of actors typically involved. Academic courses in project management could focus on adaptive strategies and leadership development. Future policies should advise and support a standardization of study management tools. Policies should also support professional development opportunities for existing researchers in the field of management, and adequate funding, given the more rigorous approach needed for interdisciplinary studies. Then there is the challenge of timing – the time it takes to develop positive communication, mutual goals and a project protocol before the study even starts. This can translate to questioning how a current project could be expanded or adjusted to fit a new grant call, thereby using established project resources and working relationships. Awasthy et al. summarize best practices for continued collaboration with industry [19]. Some of their suggestions include building mutual understanding, e.g. having people in leadership positions with an understanding of both academic and business realms who can also fill the role of mediator between the two, referred to as “boundary spanners” by Thune [19, 55]. Other best practices include reducing resource and financial costs associated with interactions, having a policy to address conflicts as they arise and continuous collaboration from designing the research questions to applying the outcomes [19].

While we have argued that we reached saturation based on themes, we do acknowledge that saturation of themes and meaning are different [56]. This study is the first of its kind, to the best of our knowledge. The identification of these themes is a solid starting point to structure future interview studies of researchers about their study management practices, in both digitally enabled and any other health study. No two studies were similar enough to compare directly or claim saturation of meaning, only breadth of insights within themes. One study, one set of research questions and one iteration of this avenue of inquiry is not enough. Saturation of meaning can be established via the repetition of such interview studies, each with a more micro-focus and in-depth inquiry into, e.g. certain themes or via more homogenous informant groups. In doing so, there is potential to more comprehensively and thoroughly understand how human, technology and health research co-exist to generate new knowledge in the field of digital health.

Conclusion

As budding researchers, many are taught the objective aspects of a study – a checklist of ethical approvals, protocols, and publications. However, study management cannot rely only on following a checklist. It is more about keeping the set of proverbial plates spinning. This interview study presents new knowledge on the realities of working in collaborative, digitally enabled health research studies. The interviews demonstrate the evolving scope of digital health studies to balance the agendas of industry, meant to develop consumer products, and those of scientific research, meant to explore the unknown and share knowledge. This evolution seems to be one of trial and error – of trying different or a combination of work cultures in collaborative teams, digital tools for remote communication and coordination of study tasks and a combination of technologies and strategies to manage the overall study.

Through this interview study, we have hopefully encouraged even more investigation and conversation about comprehensive analyses of study performance – not only the outcomes of the intervention but the performance of activities behind the scenes. We intend for this knowledge to be especially available to those in the academic setting. It is only education on topics such as interdisciplinary collaboration, industry partnerships, concepts of team management, and technologies that may help with the organization and planning of studies.