Virtual and augmented reality gamification of visuospatial neglect treatment: therapists’ user experience

Abstract

Background

Visuospatial neglect (VSN) is a cognitive disorder after stroke in which patients fail to consciously process and interact with contralesional stimuli. Visual Scanning Training (VST) is the recommended treatment in clinical guidelines. At the moment, several mixed reality versions of Visual Scanning Training (VST) are being developed. The aim of this study was to explore the opinions of end-users (i.e., therapists) on the use of Virtual Reality (VR) and Augmented Reality (AR) in VSN treatment.

Methods

Therapists played one VR and two AR Serious Games, and subsequently filled out a questionnaire on User Experience, Usability, and Implementation.

Results

Sixteen therapists (psychologists, occupational, speech, and physiotherapists) played the games, thirteen of them evaluated the games. Therapists saw great potential in all three games, yet there was room for improvement on the level of usability, especially for tailoring the games to the patient’s needs. Therapists’ opinions were comparable between VR and AR Serious Games. For implementation, therapists stressed the urgency of clear guidelines and instructions.

Discussion

Even though VR/AR technology is promising for VSN treatment, there is no one-size-fits-all applicability. It may thus be crucial to move towards a plethora of training environments rather than a single standardized mixed reality neglect treatment.

Conclusion

As therapists see the potential value of mixed reality, it remains important to investigate the efficacy of AR and VR training tools.

Introduction

Visuospatial neglect (VSN) is prevalent after stroke in an estimated 30–90% of cases, depending on the diagnostic tests used and the time post-stroke onset [1, 2]. VSN is a cognitive syndrome in which the core deficit is impairment of lateralized attention. VSN can be observed in daily life as a failure – or being much slower - to find belongings at one side of space, or a failure to avoid bumping into objects at one side of space. VSN has a negative impact on recovery in general, not only on the functional level (e.g., attention, motor impairment) [3], but also on activities of daily living (e.g., grooming, cooking) [4], participation in society (e.g., working, leisure activities) [5,6,7], and informal caregiver’s burden [8]. Therefore, there is consensus that early and better assessment, and tailor-made treatment for VSN is mandatory. Visual Scanning Training (VST) is the most recommended treatment for patients admitted for inpatient rehabilitation, according to the European Federation of the Neurological Societies guidelines of cognitive rehabilitation [9]. During VST, the patient is trained to make eye and head movements towards the contralesional side of space and to display compensatory exploratory behavior, by means of cues and feedback [10,11,12]. Tasks that are mostly used to train visual scanning are reading, letter search, and scanning pictures [12]. Although VST is frequently used, there is need for better transfer of trained scanning behavior to daily life situations. A promising way to improve VST is to pair VST with Augmented Reality (AR) and Virtual Reality (VR) Serious Games. There is growing evidence for the applicability of AR and VR for cognitive rehabilitation, due to the ability of monitoring of patient’s performance, generating more objective measurements, and training in a dynamic and more ecologically valid fashion [13,14,15,16,17]. Serious Games are videogames that are not solely meant for entertainment, but have the additional purpose of, for instance, providing information or training [18]. These games are praised for their motivational and reward elements that can benefit rehabilitation by enhancing therapy compliance [19, 20]. Although more research is required to establish the added value of Serious Games, a recent systematic review revealed positive effects of Serious Games on the rehabilitation of attention in various patient populations [21].

The focus of the current study was to evaluate the potential applicability and usability of three novel VR and AR Serious Games, developed to be used during VST. Developing new methods and/or (technological) tools for application in rehabilitation is booming. More intensive treatment is needed, especially focusing on interactions in dynamic settings, to pave the way to practice and use compensation strategies in settings that resemble daily life. Notwithstanding the fact that for implementation in clinical care, efficacy and effectiveness need to be proven, one aspect crucial for clinical implementation is often overlooked: the feasibility and user-experience of (professional) end-users. Input from end-users is required as one of the implementation tactics [22, 23] to move from the experimental setting towards the clinic [24, 25]. Therefore, the Involvement Matrix [26] and the Design Thinking model [27, 28] were utilized wherein end-users are involved throughout the phases of development and further development of the Serious Games.

Here, we focused on the therapists as end-users, as they are experts within the field of neglect rehabilitation, they know the current clinical guidelines, and have insight into the strengths/weaknesses of conventional therapy [29]. Suggestions and concerns raised by therapists can be considered during the ongoing development and implementation of the games and give way for future studies [30].

Hence, the aim of this study was to scrutinize the opinions, concerns, and suggestions on the use of VR and AR Serious Games for the rehabilitation of neglect by therapists. The first sub-aim was to explore the usability, so that patients can operate the games independently or with the help of a therapist/informal caregiver. Given that VR and AR technologies have much in common, yet are different in usage and possibilities, the second sub-aim was to explore potential differences between AR and VR with respect to usability and implementation. The third sub-aim was to investigate the requirements for implementation of the VR and AR Serious Games in the clinic.

Methods

Participants

Therapists recruited for this study worked at De Hoogstraat Rehabilitation Center Utrecht, The Netherlands. Therapists had to be involved in assessing and/or treating patients with VSN to be included in this study. All participants that have participated gave their informed consent. The study was approved by the Medical Ethics Committee at University Medical Center Utrecht and the Ethical Committee of De Hoogstraat Rehabilitation Center (no. 21/706).

Procedure, Serious Games, and outcome measures

The therapists were invited to play the games (Fig. 1) during a group session in a large room. Each game was supervised by one researcher (EB, HH, or JB). Researchers helped with placing the headsets, gave instructions, and answered any questions the therapists had. Therapists could choose which games they played and in which order. The gaming sessions were approximately 10–15 min per game (AR Virtual Museum, VR HEMIRehApp, AR Balloon Popping; described below) per therapist. After playing the games, the therapists could fill out an online survey assessing their experience with, and thoughts about each game. A weekly reminder was sent for a duration of three weeks when the survey was not submitted. The survey was conducted with Qualtrics software [31] and took approximately 8 min per game. The outcomes were examined on group level.

Fig. 1

Display of each game, from left to right: AR Virtual Museum, VR HEMIRehApp, AR Balloon Popping

AR Virtual Museum

For this AR game a ‘Microsoft Hololens 1’™ (Microsoft), with 1268px x 720px resolution per eye, 60 Hz refresh rate, 35° field of view diagonal was used. The purpose of the Virtual Museum is to stimulate visual search while moving through a room. Virtual paintings are projected on walls in the real world. Participants are instructed to move around the room and search for the paintings. When a painting is found, the image turns into a short video clip. After the clip, the next painting could be searched for. A detailed description of the Virtual Museum and the development process is described in an earlier article by Bakker et al. [28].

VR HEMIRehApp

A Head Mounted Display ‘Oculus Rift’™ (Meta Quest) CV1 (1080 × 1200 pixels resolution per eye; 90 Hz refresh rate; 110° field of view diagonal) with handheld Oculus controllers was used. The aim of this VR game is to retrain spatial attention orientation towards the contralesional side of space [32]. Participants are presented with a virtual farm and are instructed to help their neighbor on this farm. Tasks are to harvest vegetables, fishing, or to feed deer. There are multiple levels in the game, with different tasks and daylight. For demonstration purposes, targets were presented uniformly across the visual field and only a single level was shown. Instead of the in-game tutorial, the researcher provided the instructions verbally. The game was originally designed to target right-hemispheric stroke patients with intact language abilities [32]. The validity and usability of the game have been explored in a pilot study wherein HEMIRehApp corresponded to neglect outcomes in computerized cancellation tasks and establishing good usability [33].

AR Balloon Popping

The AR game was designed by Holomoves [34]. A ‘Hololens 2’™ (Microsoft), 2048px x 1080px resolution per eye; 60 Hz refresh rate; 52° field of view diagonal, was used. Although AR Balloon Popping was initially designed to motivate patients to start moving sooner and more frequently during admission in hospital, the design of the game was extremely well suited for assessment and/or treatment of visuospatial neglect. The assignment and layout of the game resembles a conventional cancellation task, with balloons being projected left and right of the participant. The game’s current purpose was therefore to train visual search with physical exercise by evoking movement. Virtual balloons were projected in the space surrounding the player. Participants were instructed to move around and search for the balloons. When found, balloons could be ‘popped’ with one’s finger. Auditory cues (female voice, instructing where to search) and visual cues (green arrows indicating the search direction) were provided. Therapists played three different games: an introductory game where all the balloons had to be popped, image pairings by pairing the balloons with the same image, and word searching by forming the correct word with the letters on the balloons. The playing duration was about 6 min.

Survey

A survey was designed by the authors which consisted of quantitative and qualitative questions and statements, aimed to evaluate the therapist’s opinions regarding usability and implementation of the games (see Additional file 1 for the survey). The questions were largely based on the System Usability Scale [35]. Given that this is a quantitative scale for quick evaluations, we have extended our survey with specific in depth questions targeting VST, Preference, Usability and Implementation. First, participants filled out their profession and experience with VSN. Next, they were asked to rate their daily experience with different types of technology (i.e., computer, smartphone, virtual reality, augmented reality, consoles) on a 5-point scale ranging from 1 (strongly inexperienced) to 5 (strongly experienced). Then, several general aspects of the game were rated (i.e., preferability, difficulty, applicability, and duration), and aspects of the game content (i.e., graphics, clarity of instructions, variation in tasks, sound, and theme) on a 5-point scale ranging from 1 (very poor) to 5 (very good). Finally, nine mixed questions (i.e., multiple choice questions with an open textbox, or open questions) on user experience were included.

Analyses

To gain more insight on the usability and implementation of the Serious Games and differences/similarities in AR/VR, both qualitative and quantitative data were analyzed. Per game, descriptive data was provided for all multiple-choice questions. Some multiple-choice questions were followed up by open questions, leaving room for therapists to elaborate on the subject. Qualitative data was then extracted from Qualtrics [31] output and further analyzed with NVivo software version 12 [36]. Consecutively, a coding method was used according to Boeije et al. [37]. Following this method, the researchers (EB and JB) independently applied the open coding procedure for each question from the survey by labeling the data to create a list of significant codes. The coding scheme was developed based on the topics that were mentioned most by the therapists. Data saturation was met, as there was no added information with an increase in respondents. Subsequently, the themes were listed in a coding scheme defining each theme with a description, and a significant or prototype statement/answer from the dataset. The next step was to code the themes axially, wherein the themes were clustered in main and subthemes. Thus, creating a compact coding scheme. After five iterations there was consensus among the authors EB, HH and TN. The final coding scheme was divided into the categories positive aspects, negative aspects, suggestions for adjustment of the games and therapists’ needs. Then selective coding was applied [37] by arranging the themes to answer the sub-questions of the current study on implementation, usability, and comparisons of the three games.

Results

Demographics and background of therapists

In total, 16 therapists (87% female; age: M = 43 years, SD: 11.96, range 25–61 years) participated in the study. Of this group, 53.84% were occupational therapists, 30.76% were psychologists, one person was a physiotherapist, and one was a speech therapist. Of all participants, 13 of them filled out the survey (84.6% within 2 weeks; 15.4% between 5–8 weeks). Not everyone evaluated all three games: 38.5% evaluated all three of the Serious Games, 23% evaluated two, and 38.5% evaluated only a single game. Regarding their experience with VSN, 84.6% were involved in both assessment and treatment of VSN, and 15.4% in treatment only. No therapist reported being involved in neuropsychological assessment only. Figure 2 shows the group was strongly experienced with computers and smartphones (38–46%) and reported to be strongly inexperienced with AR (92%), VR (53%), and with consoles (46%; e.g., Nintendo, Play Station).

Fig. 2

Experience with technology. Note: Therapists’ (n = 13) ratings of their experience with technology (i.e., consoles, AR, VR, smartphone, computer) on a scale of 1 = strongly inexperienced to 5 = strongly experienced

Opinions and user experience

The coding scheme (Additional file 2) was composed of four categories: Positive aspects of the games, Negative aspects of the games (without further specifications or solutions), Suggestions and solutions on how to improve the games, and Need for insights in game performance.

Usability

General positive remarks (Fig. 3A) were given by 30.8% of the therapists (e.g., “Nice game that invites you to be active” or “Good that there is a 360° view in the games”). Zooming in on applicability and implementation, 54.0% of the therapists mentioned the games were definitely suitable for neglect rehabilitation (Fig. 3A). More specifically, 23.0% indicated that the games contained motivational aspects to practice VST. Not only were the games regarded as suitable for training, therapists stated that more purposes could be met with the games (30.8%), stressing a role within VSN assessment. Therapists also mentioned potential for other patient groups such as patients with hemianopia (15.0%), other treatment, such as motor rehabilitation (15.4%), and finally, psychoeducation for patients and caregivers (15.4%). With respect to clinical use, 15.4% of the therapists indicated that patients could play the Serious Games autonomously, whereas 23.0% stressed supervised gameplay.

Fig. 3

The coding scheme reflected in percentages of therapists (n = 13) that mentioned: A Positive aspects, B Negative aspects, C Suggestions, and their D need for insights in game performance

Obviously, there were also limitations to the games. Therapists critiqued some aspects of the games without offering solutions immediately, as can be found in Fig. 3B. It shows that 53.8% of the therapist had general remarks on the need for refining the games (e.g., “The graphics could be improved.” or “The music was too loud.”). The main aspects to be improved in future versions were motivational features (23.0%; especially when the games are going to be played multiple times during VST), direct and delayed feedback (23.8%; more specific and more frequently), and rewards after (good) performance during training (7.7%), as well as a lack in difficulty/degree of challenge (23.0%). Some mentioned “The tasks could be too easy for the patients” and “May even be a bit boring to keep on playing”.

Next to the stand-alone remarks on positive and negative aspects of the games, therapists offered more constructive suggestions and solutions for further development (Fig. 3C). Approximately 30% questioned the versatility, or the extent to which each game could be accessible and suitable for a heterogeneous clinical population. The games would be more applicable when the games were adaptive to patients with cognitive or motor disabilities. There were concerns about patients with hypersensitivity to sensory input (e.g., “Due to the bright colors and the music there are less possibilities for patients with problems in the processing of sensory input”). It was therefore suggested to build in more flexibility in the games (30.7%), so that the sounds and contrast may be adjusted to the patients’ needs. Moreover, flexibility and variation accounted for the possibility to choose the theme of the game as well. Approximately 23% mentioned suggestions specifically for game content and themes. Patients could be motivated when the themes could be selected that fit their interests (e.g., sports; nature), or where for example, videos for different time periods are available to accommodate patients across a diverse age range. Moreover, other in-game adjustments that were mentioned covered a wider range of game features (23.1%). These included difficulty/challenges, for example by spreading the targets further apart. Other features encompassed the technique’s potential, such as using the surroundings in the AR games. Therapists also expressed their needs for in-game improvements. Suggestions for feedback and reward were mentioned explicitly (23.1%). There was a need for feedback on the search strategy the patient displays, and positive reinforcement of correct trials. Yet there was no consensus on the exact timing: directly during or after a specific trial or assignment, or after completion of a whole subgame. Related to this information, 30.8% of the therapists expressed the necessity of insight in information on patient’s game performance in order to keep track of therapy progression (Fig. 3D). The insights encompassed accuracy, duration, and information on search strategy, including movement towards targets and gaze directions. Interestingly, 23.1% mentioned other innovative outcome measures, such as a combination of accuracy versus target location.

Differences and similarities between VR and AR Serious Games

All in all, there were no obvious differences in strengths and need for improvements between the AR and VR Serious Games (Table 1). Figure 4 shows that the general aspects per game and the game content were also comparable. Differences were mostly at the level of the individual games, irrespective of the platform (AR/VR). Importantly, with respect to the implications and potential clinical implementation, all games would potentially be recommended to therapists’ colleagues in the field, either as is or after adaptations in line with the suggestions.

Table 1 Therapist’s answers on the multiple-choice questions, split per game

Fig. 4

Therapist’s ratings of general aspects of the game and game content, split per game. Note: General aspects of the game (i.e., preferability, difficulty, applicability, and duration), Game content (i.e., graphics, clarity of instructions, variation in tasks, sound, and theme) were rated on a 5-point scale ranging from 1 (very poor) to 5 (very good)

Implementation

Figure 3C displays that 42.2% of the therapists indicated that, prior to implementation, there was an urgent need for guidance (e.g., training, education, protocols) on how to play the games, and handling the equipment (e.g., storage and reservations). More detailed and clear instructions were also requested for each Serious Game (40% to 62.5%; Table 1 section B). More specifically, different protocols for therapists and patients seemed to be a prerequisite for implementation in a clinical setting. For therapists, a manual with text and images aiming to give the instructions themselves, including a section on the different gaming possibilities was suggested. A similar manual would be recommended for therapy with the help of an informal caregiver (Table 1 section C). For patients, an interactive manual could be integrated in the game. In other words, patients could ‘learn to play the game’ (e.g., to navigate, equip tools, solve puzzles) within the game context, without recourse to direct instruction outside the game. Furthermore, protocols were considered regarding the use of the hardware for training sessions and handling the hardware (e.g., storage and reservations). Considering practical matters, costs for the equipment and potential license fees were mentioned frequently. There was a request for more insight into the total costs of the use and maintenance of the hardware and software.

Discussion

The overall aim of the study was to contribute to the development of more intensive treatment for VSN by means of novel technology to accompany VST. Hereby taking into account interactions in dynamic settings to the practice and use of compensatory strategies in settings that resemble daily life. This study explored the user experience of therapists on three Serious Games dedicated to support VST, the currently recommended treatment for VSN in clinical guidelines. The usability of the games, potential differences/similarities amongst the techniques, and the requirements for implementation were explored.

Overall, therapists were positive about the three games and saw merit in the application of those Serious Games in clinical practice. They even mentioned additional applications, such as VST for patients with hemianopia, patients playing together, as a motivating game for motor rehabilitation, as intuitive psychoeducation to patients and informal caregivers (due to the interactive gameplay), and as means for more detailed outcome measures. However, the Serious Games were not ready to be implemented in clinical care immediately. Therapists emphasized the importance of (1) clear instructions (2) a detailed manual for professionals, (3) good in-game tutorials and instructions for patients, (4) tailoring the levels and content to meet the needs of individual patients, (5) clear feedback and reward features, and (6) insight in gaming performance of patients. The strengths and prerequisites were comparable between Virtual Reality and Augmented Reality Serious Games used in this study.

Strengths and limitations

For innovations in healthcare to be implemented and not linger at the stage of research, it is acknowledged that the end-users should be involved in the design and development process. In all the Serious Games used in this study, end-users were already part of the team in development and redesign by means of The Involvement Matrix [26] and the Design Thinking model [27, 28]. Here, we were able to receive feedback from a multidisciplinary team dedicated to inpatient neurorehabilitation, with all the primary disciplines represented. Data saturation was met, enabling us to set specific adaptations and improvements for future research towards actual implementation.

A potential limitation was the single site design. The rehabilitation center - and the therapists - is leading in the field of scientific research, and (implementation of) innovative solutions in healthcare and has ample experience with research, innovation, and various technology (e.g., computerized assessment and the use of tablets during training). Consequently, the therapists of this center may have a more positive view on usability and implementation in comparison to other therapists. On the other hand, although the therapists at this site are frequently involved in innovation projects, they did report having little experience with AR and VR before participating in this study. With respect to literature, our demographic data and sample size did not differ from similar studies, for instance, Morse et al. [30], Bakker et al. [28], Tobler-Ammann et al. [38]. A multicenter study with the next versions – based on the current feedback – and both patients and professionals would give important insights in the generalizability of opinions and feedback. Another point worth making is that not all therapists filled out the questionnaire for each game. However, for this study it was not intended to use a within subject design and the corresponding analyses were not applied to compare the three games. Related to the response rate, the potential effect of non-rating could not be traced. Given data saturation, different feedback or opinions on the games were not expected. There is also the time taken to fill out the questionnaire after playing the games. As most of the therapists filled out the survey within 2 weeks, two outliers with a longer duration were found as well. These were included in this study due to the relatively small sample size. Moreover, there was no great variability found within our dataset that could be explained by the two outliers.

Future research and clinical implications

All things considered, not much is known about the implementation of Serious Games in the clinic, the user experience, feasibility, along with efficacy of the games. Apart from AR Virtual Museum and VR HEMIRehApp, AR Balloon Popping was not designed for implementation for VST, but for motivating patients with various medical backgrounds to become more active. It is therefore noteworthy that all three games were reported as a useful addition to conventional therapy for VSN. The latter indicates the possibilities of exnovation through recycling single purpose technology.

Before implementing the games, concerns raised at the level of practicality and usability must be solved as well. Practical concerns encompassed setting up protocols for embedding the games in (group) therapy, a protocol for operating the games, and one for handling the equipment. Especially when the goal is to let the patients play the games on their own to help make better use of therapy time, there should be proper education or a written manual on how to play the games. The latter is in line with the findings of Morse et al. [30] who concluded that the clarity of instructions could be a barrier, according to the end-users. They advise that instructions cannot be too lengthy, as this will complicate the understanding of the aims and steps. In a similar study, Tobler-Ammann et al. [38] indeed found that their short instructions were positively reviewed by the end-users. The current study could add that the manuals for therapists need a different approach from the manual for patients. Patients require in-game tutorials/instructions, whereas therapists prefer a written manual for game use and interpretation. An additional practical concern was the management of costs for equipment and software fees accompanied by all three games, although it is important to note that clinicians did not receive information about these costs. Powell et al. [39] also advise to increase economic evaluations in the implementation process to demonstrate the cost-effectiveness of the intervention. With respect to usability, the games were multipurposed, in that the games are suitable for VSN training, may be useful for assessment of VSN, and for training other pathologies. However, there were suggestions on the versatility of the games, wherein accessibility for a heterogeneous clinical population must be considered. Cognitive and motor abilities and hypersensitivity to sensory input were all mentioned to be considered as potential factors interfering with the usability of the games. Especially the AR Virtual Museum might be challenging for patients with hemiparesis as - in the main setting -, the game primarily encourages patients to move around in the room. This might affect patients with impairments in balance, who have general problems with moving around in their environment [40]. The other Serious Games might be more suitable, for they can be played sitting or even lying down. Adaptation of the games to the information processing abilities of patients, patients’ mobility and in-game flexibility in visual and auditory output are thus recommended. Other forms of adverse effects like cybersickness [41] were not addressed by the therapists. Possibly therapists did not experience any cybersickness and did therefore not come up with such a potential adverse effect. In earlier studies on VR HEMIRehApp and AR Virtual Museum, participants did not report cybersickness [28, 33].

There were mixed experiences on the motivational aspects and challenges in the games. The games were thought not to pique everyone’s interest. Contrastingly it was mentioned that the games were motivational for VST. First, a solution would be to present options to personalize the game content, so that a patient can choose from several themes (e.g., a theme with animals or historical facts). Second, in-game difficulty adaptations to the patient’s performance would be recommended as well. Not only adaptations to patients’ performances were important suggestions. Therapists also pointed out the need for gaining insights in performance on the levels of accuracy, duration, search strategies, and a combination of these outcome measures. Although not new to scientific research, there is clearly a need for fine grained recovery measures for VST in healthcare.

Conclusion

Therapist’s opinions were a valuable addition in the journey for innovating therapy for VSN and may be applicable to a broader category of novel techniques for rehabilitation. Apart from the therapists, expertise of the patient group, and their informal caregivers could be considered as a next step for further development of the VR and AR Serious Games.