Ethnic representation within virtual reality: a co-design study in a forensic youth care setting

Abstract

Background

Virtual reality (VR) is one of the most cutting-edge digital technologies currently used to extend and adapt mental health interventions. When designing therapeutic VR-applications, bias related to representational and behavioral features can be introduced. Specific bias, that could result from ethnic stereotyping, should be averted. However, structured debates and clear guidelines on how to deal with such bias are lacking.

Objective

In the current paper we describe our iterative, co-design process to develop an augmented version of a 360-degree VR-video that is used in the Street Temptations intervention. In this intervention, aimed at adolescents with disruptive behavior problems, the VR-video provides adolescents with an emotionally engaging situation and forms the basis of Street Temptations’ perspective-taking exercises. By presenting a detailed example of our VR developmental process, specifically focusing on ethnic representation in this virtual environment and related ethical aspects, we aim to positively contribute to the creation of ethically sound therapeutic VR-applications.

Method

We adopted an experience-based co-design approach. The process started by assembling professionals’ experiences with the original VR-video. Following, we conducted four co-design groups with adolescents, young adults, and professionals within the forensic youth care system, as experiential experts, to collaboratively develop an enhanced VR-scenario for the Street Temptations intervention. In total, ten adolescents, four young adults and four professionals participated. The co-design groups were alternated with sessions with a screenwriter, to reflect on the results of the co-design groups. In this way, we enabled continuous feedback on and design of the ameliorated scenario and let development and data collection reinforce each other.

Results

The most important adjustments that were brought forward included explicit incorporation of diversity, establishing a clear storyline for the scenario, keeping the content close to reality, casting actors in a way to avoid obvious stigmatization, and using only names for the characters that are commonly used across different cultural and ethnic backgrounds.

Discussion

By presenting our developmental process, we give a detailed example of how to reflectively develop ethically sound therapeutic VR-content within a forensic youth care setting. We demonstrated that including stakeholders can help in preventing socially irresponsible and conceivably hurtful stereotypes. By describing the process in detail, we hope to contribute to the responsible use of technology within mental health care.

Introduction

Digital technology is increasingly used to extend and adapt evidence-based face-to-face psychotherapy [1]. The resulting digital health interventions, e.g. web-based interventions and apps [2], seem to pose several advantages over more traditional forms of therapy, such as applicability in real-life contexts, accessibility, personalization, and facilitation of additional practice [3,4,5]. One of the most cutting-edge technologies currently used in digital mental health interventions may be virtual reality (VR) [6].

With VR, it becomes possible to replace one’s physical environment with computer-generated experiences [7]. When applied effectively, VR has the potential to generate powerful and effective learning opportunities [7], creating new possibilities when it comes to mental health care [6]. Initially used in the treatment of phobic disorders, VR-technology has currently been implemented across a range of mental health disorders and problems [8]. Promising results concerning VR-interventions for mental health problems in children and adolescents are emerging, although these interventions are to date less extensively studied compared to interventions in adults [9].

Already in the last century, important ethical issues surrounding VR were highlighted [10]. Going beyond specific uses or areas of application, Brey [10] addressed two essential ethical aspects of VR-technology, i.e. representational and behavioral features. Representational features of VR relate to the way in which reality is represented in a virtual environment [10]. As explained in Brey’s essay, these features form a spectrum. At one extreme are the most detailed representations of reality, at the other are virtual environments and characters that do not intend to have any link with reality. Between these two extremes, all kinds of variations regarding the degree of realism can emerge.

Next to the representational features, behavioral features of VR relate to the behaviors that are visible or made available in virtual environments [10]. For these environments, the possibilities are in principle endless. Therefore, it is possible to depict or enable behaviors that would be unethical or immoral in the real world. This potentially leads to ethical issues. Violence, for instance, poses possible concerns in light of the behavioral and emotional effects of VR that have been demonstrated [7]. Violent behavioral features may, however, be pertinent in certain therapeutic VR-applications.

Meanwhile, ethical aspects of VR and recommendations for ethical VR development have been discussed, focusing on, for example, legal issues, physical discomforts, developmental worries, and moral concerns (e.g., [11,12,13]). However, representational and behavioral features, specifically regarding representational and behavioral bias as a result of ethnic stereotypes, have not been addressed often, although some potential issues have been identified [14]. Additionally, a few recommendations for best practices have been formulated and the necessity of culturally appropriate VR has been stated (e.g., [15,16,17]). The majority of these recommendations focus on process-oriented factors, for example, working together with ethnic minority stakeholders when designing and developing VR-applications [15]. Nevertheless, it remains unclear how such a developmental process could successfully be organized and what steps would be involved. In the current paper, we will focus on the ethical aspects regarding representation and behavior in the process of co-designing a therapeutic VR-application, specifically in relation to ethnic stereotypes.

Representational and behavioral features should receive particular attention when it comes to therapeutic or educational VR-applications. After all, such applications specifically aim to modify peoples’ real-life behaviors and beliefs [10]. Consequently, just because VR’s possibilities are endless and virtual experiences can be seen as actual real experiences rather than merely a media experience, it is argued that the development of VR-experiences entails a certain moral responsibility [7, 10, 18].

Practical aspects such as time, money and expertise may lead to pragmatic decisions regarding representational and behavioral features when designing and developing a VR-application. Whereas Brey [10] states that such decisions may lead to biased representations or misrepresentations, in his follow-up article to Brey’s essay, Ford [18] even states that mis- or biased representations will be inevitably present in every virtual environment created. Therefore, we need to consciously choose which biases are implemented in a virtual environment and be aware of the implications associated, so that we can act in an informed and responsible manner [18]. In doing so, socially irresponsible and conceivably hurtful stereotypes should be averted. This becomes even more important when potentially unethical or immoral behavioral features are present, such as the aforementioned violence.

Specific biases could lie in the use of stereotypes in the representation of people and events, which may be rooted in, for example, ethnic stereotypes [10, 14]. Considering the legacy and ongoing reality of (systemic) racism and discrimination across the globe [19], these stereotypes should especially be avoided. In combination with the potential harmful effects of perceived racial discrimination on psychological well-being [20, 21], and the fact that bias due to racial discrimination has been shown present in virtual environments [15, 22], being able to act in an informed and responsible manner when it comes to ethnic stereotypes is vital.

In order to act in an informed and responsible manner concerning ethnic stereotypes ourselves, we adopted a co-design approach to improve the VR-augmented intervention we are currently developing and evaluating. The intervention, titled Street Temptations, aims to increase the treatment responsivity of adolescents with disruptive behavior problems. A 360-degree VR-video is used to provide a vivid and emotionally engaging representation of a socially challenging situation, namely a street fight resulting from peer pressure. This VR-scenario is used as an integral starting point of Street Temptations’ sessions, and the effect of the VR-scenario is transferred into playful and dynamic exercises to encourage adolescents to take different perspectives and reflect on both their own behavior as on that of others. Because a violent VR-scenario is used as the exercises’ starting point, we find ourselves at the complex interface of possible ethnic stereotypes and the presentation of potentially unethical or immoral behavioral features as discussed above. For a more detailed description of Street Temptations, please see Klein Schaarsberg et al. [23] and Klein Schaarsberg et al. [24].

When training professionals to execute Street Temptations in order to evaluate the effects of the intervention, it became clear that the existing video could encourage ethnic stereotyping. We realized ethnic representation had not been an explicit topic of discussion during development of the original video. As we regarded explicit discussion of this topic prior to further studying of Street Temptations essential to act in a more informed manner, we stopped the study and decided to ameliorate the VR-video. Our primary goal was to create possibilities for making conscious and deliberate choices regarding ethnic representation in our VR-scenario, referring to the ethnic backgrounds or identities of the characters and their roles in the video. These choices could, for example, relate to skin color, hairstyle, and names of the characters.

To make deliberate choices regarding ethnic representation, we engaged in collaboration with targeted end-users and put their views and experiences central to the change process. More specifically, we choose to adopt a co-design approach. Central to co-design is that users are seen as “experts of their experiences”, and to explicitly incorporate their experiences in a process of active collaboration [25, 26]. Therefore, co-design involves a joint exploration of needs, resulting in a joint creation of solutions [27]. By adopting a co-design approach, we aimed for a deeper and richer understanding of users’ thoughts and feelings about how ethnic representation in VR should be improved, for example, moving beyond mere consultation, enabling this understanding to augment the VR-content [26]. Moreover, co-design is proposed to be beneficial for the creation of more engaging and satisfying interventions [26].

In the current paper we describe our co-design process to develop an augmented version of the first Street Temptations VR-video, specifically focusing on how we could avoid possible adverse ethnic stereotypes. Together with stakeholders, we discussed options to work with 360-degree VR in an ethically responsible manner, in line with the format of the original VR-video. By presenting this detailed example of our co-design process, specifically focusing on ethnic representation in virtual environments, we strive to contribute to an ethically sound landscape of therapeutic VR-applications.

Methods

Approach

Our approach is based on a scientific methodology framework by an international working group (VR-CORE; Virtual Reality Clinical Outcomes Research Experts), to guide therapeutic VR development and research specifically [28]. In addition to this framework, we follow the recommendations by Taylor et al. [15], especially related to interracial interactions in virtual environments. Emphasized in both reports, is incorporating the voices of the people at stake.

Our current study belongs to the first phase of VR-CORE’s framework (VR1), in which content is developed through collaboration with patient and provider end-users. The working group identified three key design principles in their framework, based on the principles of human-centered design (see Table 1): promoting empathy, team collaboration, and continuous user feedback [28]. To incorporate these three key principles, we operationalized the VR-CORE framework into an iterative co-design approach [26, 29] to augment our VR-content. We adopted an experience-based co-design (EBCD) approach more specifically, which we adapted to a feasible procedure given the contextual constraints. EBCD originates from healthcare service and quality improvement, but also shows potential for intervention design [30].

Table 1 VR-CORE’s key design principles of the first phase

Experience-based co-design

Essential to EBCD is the combination of an orientation on user experiences and a collaborative change process to jointly determine and design necessary improvements [31, 32]. According to a review by Green et al. [30], these essential parts are translated into two phases of EBCD: an experience gathering phase and a co-design phase in which improvements are designed based on the experiences identified. Fundamental to the latter co-design phase is a certain degree of co-deciding, ensuring that participants do not just make recommendations [32, 33]. Ultimately, the two phases are translated into different stages, which together form the EBCD cycle [32, 34].

In our case, experience gathering in the first place resulted from training the professionals in executing Street Temptations in order to prepare them to participate in the effectiveness study we were going to conduct. Based on the professionals’ experiences during the use of the original VR-video, it became clear that the video could encourage ethnic stereotyping. After receiving this feedback, two separate meetings to collectively discuss their experiences and feedback followed. The outcomes were of such concern that we did not consider further collection of experiences necessary nor appropriate before looking into potential improvements. Therefore, we did not make further use of common EBCD activities for this phase, such as filmed narrative-based interviews [32]. However, we did gather additional experiences during the further process of co-designing, by reflecting with users on new versions of the VR-scenario.

During the phase of co-design, one of the important stages is to work in small co-design groups centered around the priorities previously identified [30, 32]. We operationalized this stage into an iterative process centered around our research questions, identified on the basis of the experiences of the professionals (see Fig. 1). To gather as much input as possible in a relatively short time period, we chose to organize multiple unmixed co-design groups with different stakeholders. Although it is recommended to mix groups [35], we foresaw practical obstacles to establish this successfully in our current study. Additionally, we wanted to limit the burden on participants, so we intentionally opted for one time co-design groups with each group of participants. We did adhere to the typical involvement of 4–6 participants per group [32]. This group size is also recommended when tackling sensitive or complex issues [36], which we were expected to do, e.g. stereotyping and stigmatization. Furthermore, a separate interview was done with one of the professionals, as this person was not able to attend the co-design group with professionals. In subsections ‘Participants’ and ‘Procedures’ we further explain our specific co-design collaboration in detail.

Fig. 1

Overview of the iterative co-design process of the new VR-video

It is important to note that we especially focused on augmenting Street Temptations’ VR-content with this co-design study and not on the complete intervention, as Street Temptations will be thoroughly evaluated by the same group of researchers during the forthcoming single-case experimental study and will be developed further accordingly [23]. Consequently, not every step of the VR-CORE methodological framework, as not every stage of the EBCD cycle, was applicable to our current study.

Furthermore, content enhancement was based on the original VR-video, which shows a group of 4 youngsters in a park. The scenario from the original video played out as follows: youngster 1 pressures youngster 2 to beat up a passerby (youngster 3), and youngster 2 obeys. In between, there are fragments shown in which youngster 2 is interviewed about why he knocked down youngster 3. The video ends with a compilation of videos from the internet of real fights between adolescents [23]. This VR-scenario is used to provide adolescents with an emotionally engaging situation they could encounter in their daily lives and forms the basis of Street Temptations’ perspective-taking exercises.

Participants

For the co-design groups, we sought collaboration with representatives of three groups: adolescents who would currently fit the target population of Street Temptations, young adults with lived experience regarding forensic youth care who used to be the target population of Street Temptations (i.e., experiential experts), and professionals that could deliver Street Temptations. We strived to include people with varying ethnic backgrounds, to help ensure that, within our single country sample, different ethnic backgrounds were represented across the participants. Additionally, both female and male participants were included. The diversity within the recruitment populations enabled including people with varying ethnic backgrounds. However, we were also dependent on practical reasons such as willingness and availability to participate. Our goal was to represent at least two different ethnic backgrounds in each co-design group. This goal was achieved based on the persons that agreed to participate. Including both female and male participants in every co-design group turned out to be impossible for the adolescent co-design groups, as they were held at gender specific treatment facilities. Additionally, only males worked at the consultancy agency we recruited the young adults from. We did not exclude persons who wished to participate, as the number of applications did not exceed the number of available spots. In addition to the stakeholders, we worked with a screenwriter, a movie director, research assistants, and an audiovisual team experienced in the production of VR-content (including a sound technician, a visual technician, an editor and a producer).

All adolescents were recruited from a youth care provider in the province of North Holland, the Netherlands. The adolescent target population for the Street Temptations VR-video shows multiple and complex externalizing and/or internalizing behavioral problems, sometimes in combination with psychiatric and addiction problems. The adolescents often come from unstable and problematic family situations and delinquent behaviour can be present. The experiential experts were recruited from a special consultancy agency, consisting of youth with lived experience in the justice system. These youth are trained to educate and advise forensic youth care professionals and other professionals in the (juvenile) justice system. Professionals were recruited from two Dutch youth care organizations. All professionals were clinically active with regard to adolescents targeted in Street Temptations. The participant composition per co-design group is described below.

Procedures

The co-design groups were held in May and June 2022, with one discussion leader and one note taker. The meetings with the adolescents and young adults were performed live, in Amsterdam. The co-design group and interview with the professionals were held as online meetings, to limit time investment. A semi-structured topic list was constructed to guide each co-design group, updated in between co-design groups, based on the status of the new scenario developments based on the results of the prior co-design group (see Supplementary material 1). We also deliberately chose semi-structured topic lists in order to allow room for discussing participants’ personal experiences, for example. All meetings lasted between 60 and 90 min. Audio recordings were made in order to transcribe the data. Adolescents were compensated for their participation with a €15 gift card from a general Dutch web shop.

At the end of each co-design group, the most important outcomes were summarized together with the participants. The summarized outcomes were discussed with the screenwriter, during separate reflection sessions in between the co-design groups, as displayed in Fig. 1. A digital mind mapping tool (Miro, [37]) was used to synthesize outcomes, feedback and prototype versions during the reflection sessions. First, the outcomes of each co-design group were discussed and summarized in the mind map. From the second co-design group on, the similarities and discrepancies between the outcomes of the co-design groups were reviewed, as well as the feedback received on and decisions made in relation to the prototype version that stood central in the previous co-design group. Next, the outcomes, feedback, and decisions were discussed to examine what possible changes would be necessary and whether these changes would be feasible considering other factors such as time and financial resources as well. The perspective of the screenwriter was specifically important in this, as this professional could keep track of the storyline, for example, and how it would be changed after modifications.

The co-design groups and reflection sessions with the screenwriter were alternated to enable an iterative process of feedback on new versions of the scenario and reflection on this feedback. In this way, we could process new ideas and previous feedback, improve the scenario accordingly during the next session with the screenwriter and test the results in the subsequent co-design group. Because of this iteration, the outcomes and decisions became more detailed and concrete throughout the process. Furthermore, alternation allowed us to emphatically focus on ethnic representation and related decisions during the co-design groups, whilst not having to pay attention to other, more detailed or practical oriented, aspects of the scenario as we could discuss these during the sessions with the screenwriter. More importantly, the reflection sessions in between the co-design groups helped us to bring together the different perspectives of the different groups included, providing an overarching view. Because we were going to conduct unmixed co-design groups in terms of stakeholders, which we were not able to physically bring together, we strived to bring them together by means of the reflection sessions.

Co-design group session 1

The first co-design group was conducted with the experiential experts, given the expected fruitful combination of experiential knowledge and reflective ability. Four male young adults between 20 and 26 years of age participated. Self-reported ethnicity varied between Ethiopian, Moroccan, and Surinamese descent. The co-design group started by showing the original VR-video and discussing it briefly, explicitly addressing whether participants noticed anything in particular about the video. We then introduced the aim and topic of the meeting and gave some background information about VR. After this, we let participants brainstorm to freely generate ideas for the design of a new VR-scenario, to determine whether creating an ethically sound 360-degree VR-scenario would be possible. To discuss possible compositions of actors, we used a digital database from a casting agency located in Amsterdam. The discussions required little to no managing in terms of dynamics. Everyone participated actively in the conversation and participants listened and responded naturally to each other or complemented each other on their own accord, for example. Occasionally, a specific participant was directly asked for a response. During the session with the screenwriter that followed after the first co-design group, we reflected on the results and generated a first draft of a new scenario, based on the decisions made by the participants regarding ethnic representation.

Co-design group session 2

The second co-design group was conducted with the adolescents, as the main stakeholders, and consisted of four male adolescents, aged between 13 and 19 years (accompanied by two supervisors). Self-reported ethnicity varied between Dutch, Moroccan, and Surinamese descent. The co-design group started in the same way as the first one. Thereafter, we discussed the first draft of the new scenario that was created. The digital casting agency database was used again to help discuss group compositions. Compared to co-design group 1, it was more difficult to achieve an actual group discussion. One participant mainly dominated the conversation and the other participants varied in the amount of input given. One of the youngest participants (Dutch descent) was particularly silent. He was repeatedly asked directly about his thoughts or opinion, after which he often agreed with what had been said by the others. This was also addressed by the supervisors, who also tried to get him, as well as the other participants, more involved. Simultaneously, we did not want to force anyone to respond. Afterwards, the feedback and new decisions were discussed with the screenwriter and the first draft was further enhanced, resulting in a second draft.

Co-design group session 3

In the third co-design group we opted for a more overarching reflection together with the professionals. One female and three male professionals participated. Ages varied between 26 and 45 years, self-reported ethnic descent was either Dutch or Surinamese. The co-design group, as well as the separate interview, again started with viewing and discussing the original VR-scenario. As the meetings were held online, participants viewed the video on their computer screen instead of using a VR-headset. After this, the co-design group was set up in the same way as the first one, with special attention to the differences that emerged between the first two co-design groups. We ended by discussing the second draft of the new scenario. In terms of group dynamics, this co-design group was comparable to the first co-design group. Participants had an equal share to the conversation and there were no participants that excessively dominated the conversation, for example. Because this co-design group was held online, participants could also raise their hand virtually or respond via the chat function, for example, in case they wanted to react when another participant was still talking. During the following meeting with the screenwriter, a third draft was made based on the feedback and decisions from the professionals.

Co-design group session 4

We ended the co-design groups with our main stakeholders again. Six female adolescents participated (accompanied by three supervisors). Ages were unknown due to privacy reasons. Self-reported ethnicity varied between Congolese, Dutch or Moroccan descent. This co-design group started in the same way as the previous ones. Next, we discussed the third version of the new scenario, by going through the script from start to finish. The amount of participants in this co-design group made it a little more difficult to keep track of everyone’s participation, but overall every participant played an active part in the conversation. Occasionally a participant was directly asked for their opinion, especially the supervisors were able to involve a participant at the right moment or ask further when they suspected a socially desirable answer at first. During the final session with the screenwriter, the feedback and decisions of the adolescents were used to make some last minor adjustments to the new scenario and finalize it for further developmental steps.

Development of the VR-video

After finalizing the new script, the movie director was hired. This was done after finalizing the script, as the input from the stakeholders was leading and no major adjustments by the director would be deemed necessary. After the last co-design group, the final actors were casted based on their availability on the shooting day, which was planned in July 2022. All materials for the new VR-video were shot on the planned shooting day, followed by the process of editing (see Fig. 1).

Data analysis

As described in the Procedures section, collected data was analyzed for each co-design group, and discussed in a reflective session with the screenwriter. At the end of each co-design group, we collectively summarized the results in terms of conclusions and decisions important to the VR-video. The summary served as a member check, enabling participants to comment or add, and provided crucial next steps for the design process. These steps were discussed during the reflective sessions that followed each co-design group, enabling us to relate the conclusions and decisions of the different groups to each other. As a result, we translated the co-decided outcomes into a new version of the video. This new version was discussed in a new group meeting with different stakeholders, giving rise to new conclusions and decisions regarding the video. After the iterative co-design process was finished, relevant statements from participants that were noted during the co-design groups were checked against the recordings of the sessions to select quotes to illustrate the outcomes in the paper. The quotes were translated into English and some of the quotes were slightly adjusted to improve readability.

Ethics

This study was conducted according to the principles of the World Medical Association Declaration of Helsinki [38] and in accordance with the Medical Research Involving Human Subjects act. Ethics approval for this study was obtained as an amendment of the forthcoming single-case experimental study [23], granted by the independent Medical Ethical Committee of Vrije Universiteit medical center (reference number: 2021.0114). Informed consent for participation in the co-design groups was obtained from participants, as well as from parents or a legal guardian in the case of minors.

Results

In total, ten adolescents, four experiential experts, and four youth care professionals agreed to be part of the co-design process as (non-patient) end-user stakeholders. Demographic characteristics are described above per co-design group. The outcomes of the discussions are described per co-design group, in line with the chronological order of the process shown in Fig. 1.

Co-design group session 1 – 4 experiential experts

Brainstorming regarding choices for ethically sound 360-degree VR resulted in several ideas for a new scenario. First of all, the experiential experts stated that, in our specific case of a violent situation, a clear reason behind the violence in the scenario is essential. When a clear reason is lacking, or when the reason does not come across, the violence will seem random. Consequently, a spectator might look for an explanation in relation to ethnicity:

‘It gives more, it gives more of a story about peer pressure, than about the skin color.’ ‘That is what I just wanted to say, it could also be a dark colored boy.’ ‘Yeah, you understand? But if we look at it now, it really just looks like he is beaten up because he is white.’

When the reason is clear, however, the participants stated that any person could be put in any role in the story. After all, when the reason is clear, it is about the story behind the actions and not about the ethnicity or the skin color of the people visible, for example. In addition, a story with a clear reason will help to make the content more recognizable.

The participants brought forward two concrete ideas to establish a clear and recognizable storyline. They wanted to incorporate social media into the story, and stated that a possible escalating situation could be about a conflict involving a girl that is related to other characters in the story:

‘If you would want to incorporate a girl, then maybe you could say the girl is the sister of one of those guys, or it is one of those guys’ girlfriend. And that the victim has said something wrong to that girl, or has done something wrong too, causing them to react in the way they do.’ ‘Someone stands up for her, and because he stands up for her it gets out of hand or something like that.’

In addition to a clear reason, the experiential experts stressed the absolute necessity of painting a diverse 360-degree VR picture with regard to ethnicity. They explained that reality is diverse, making diversity a prerequisite in order to make realistic content. It would not be a good idea to only show people that all look similar in ethnic background or to only show people that are more difficult to categorize regarding ethnicity. Doing this would be unrealistic and limit the extent to which viewers could identify with the characters:

‘If it would be all Dutch, it would be a bit too exaggerated, too fake. […] Yeah, for me it would be okay to have a group of friends, and that there are four guys in the group, all four with different backgrounds.’ ‘Yes, that is also really what you see nowadays.’ […] ‘That way, everybody can put themselves in the picture. Especially if you cannot keep making adjustments to the movie.’

Furthermore, the participants endorsed taking a close look at the different roles in the story. Putting someone with a non-Western background in the role of perpetrator and a person with a Western background in the role of victim would provide a very prevalent stigma. Reversing those roles, or at least making sure the role of victim is not played by a white person, helps to diminish obvious stigmatization and already paints a different picture:

‘You know, I would say, put Ismael [tan skin-color] in the role of victim for once, and incorporate Bram [white skin-color] into the group of perpetrators.’ ‘That is what I have been saying all along.’ […] ‘It is not especially just the Dutchman who gets beaten up.’ […] ‘I know plenty of Dutch boys who also’ ‘Yeah.’ ‘how shall I put it, yeah who also just show certain deviant behavior.’

Ultimately, according to the participants, it should not matter at all how the characters in the video look. It should make no difference whether a character is white or black, for example, as ethnic background should not be of relevance. The personal experiences of the experiential experts can most likely be linked to this vision. All four participants are persons of color, all of whom have had negative experiences with prejudice, stereotyping and stigmatization. This very background increases the essence of the points discussed.

Reflection with screenwriter

Based on what has been discussed during the co-design group, we ended the co-design group by collectively deciding that the following adjustments were most important: a clear reason for the escalation, diverse backgrounds, and a less stereotypical role distribution. Moreover, the reason could be found in a story that revolves around a girl and that takes social media into account.

In the first session with the screenwriter we discussed how these adjustments could be translated into the new scenario. Consequently, the following changes to the scenario were made. First, the number of characters was increased, with a group of four people and a group of three people, allowing for more diversity across the characters. Including a higher number of characters would potentially cause too much confusion, according to the screenwriter, because it becomes more difficult to make clear who is who, for example. Therefore, no more than seven people in total were included in the story. Second, the person who turns to violence was pictured as a white person, the one who becomes a victim as somebody of color, as suggested by the participants. Third, the basis for the story centered around a girl who is involved in both groups, with a picture on a social media platform being a part of the trigger for the escalation. Lastly, someone starts filming when the fight begins. A more detailed description of the first draft can be found in the topic list of co-design group 2 (Supplementary material 1).

Co-design group session 2 – 4 adolescents

The adolescents had a different view on how to act in light of diversity, compared to the young adults of the first co-design group. Firstly, they mentioned that using a diverse group of people would be best. In this way, anybody could be put in any role and it is not about one specific background, just as indicated in the first co-design group:

‘Then it is not that, say, the Moroccan boy pressures another Moroccan boy, then it is just, it could just be anyone.' ‘Then it is mixed.’ ‘Yeah.’ ‘And then there is less of a problem.’ ‘Yes, less. It is not going away completely, but it will be less.’

However, in the eyes of the adolescents, it would still be possible to stigmatize even if you use a diverse group of people. As a result, for example, it could look like you would want to misrepresent white people if you put a white person in the role of perpetrator. Eventually, working with people who all look alike regarding ethnic background, or who are less clearly classifiable by ethnicity, would prevent stigmatization the most, the adolescents stated. Nonetheless, it appeared difficult to accurately describe how the people would have to look like:

‘I would just go for the unrecognizable people.’ ‘And just to be more specific, when does somebody look unrecognizable, or how does ‘neutral’ look like?’ ‘That is hard to explain.’

When discussing the first draft of the new scenario, the adolescents were a bit hesitant regarding how often a situation like the one drafted occurs. However, the story did sound realistic. One of the adolescents was familiar with a similar situation himself and especially the supervisors emphasized that situations like these actually do occur:

‘I think it does happen, but maybe not that often.' ‘Well, if I may tell you, during one of the interviews I conducted for my own research assignment, this boy told me he knew of a situation with two guys, both belonging to a different gang, but also revolving around a girl. And she cheated on this one guy with one of the other guys, and these guys came across each other. And they ended up stabbing each other with knives.’

While looking more closely at the composition of both groups in the story and discussing details like what names would be suitable for the characters, the participants came up with the idea that religious names would probably be least potentially stigmatizing. Most religious names are, after all, used across different cultural and ethnic backgrounds:

‘Benjamin.’ ‘Adam, maybe, Adam.’ ‘Yes Adem, or Adam.’ ‘David, or something like that.’ ‘David is possible too.’ ‘These are all prophets, are they not? Biblical names actually, they do, in fact, appear both in the Koran and in the Bible.’

As described in the method section, the participant of Dutch descent gave the least input and was the most difficult to engage in the conversation. His age could have played a role in this, as he was one of the youngest participants as well. However, it also might have been the case that he had less personal experience to draw from, for example, in comparison to the participants from other ethnic backgrounds.

Reflection with screenwriter

At the end of the co-design group, we decided new names for the characters were desirable and that religion would be most helpful in this. Additionally, the confirmation regarding the idea for the new storyline was an important outcome to further develop the draft of the new scenario. However, the discrepancies regarding diversity and group composition in comparison with the first co-design group were an important outcome to reflect on during the second session with the screenwriter.

In the first co-design group it was clearly stated that diversity in backgrounds would be essential, whereas the participants from the second co-design group were much less convinced of this necessity. Finally, their conclusion was to only incorporate vagueness regarding ethnicity, without being able to specify how this vagueness would have to look like. Because the adolescents themselves were somewhat doubtful and found it difficult to indicate concretely what more ‘neutral’ persons would look like, and the young adults had been very adamant about the importance of diversity, we chose to continue with our draft that incorporated diversity among the characters. Because we had decided that religious names, used across a wide variety of backgrounds, would probably be less stigmatizing, we changed some provisional names to more universally recognized names, such as Benjamin instead of Jayden. We further worked out the characters by giving all characters a name and including some pictures of possible actors. The most important decision that was left to discuss in co-design group 3 entailed what would be best to do considering the contradictory input about diversity and neutrality. For a more detailed description of the second draft, please see the topic list of co-design group 3 (Supplementary material 1).

Co-design group session 3 – 4 professionals

In the eyes of the professionals, it was most important that adolescents can identify with the story that is being told and with what they see in the VR-video. Therefore, the professionals foremost endorsed keeping the content as close as possible to reality. More specifically, the content should primarily be kept close to adolescents’ reality and incorporate adolescents’ world of experience. Social media is an important aspect herein and should therefore definitely be incorporated in the scenario, consistent with what the experiential experts stated:

‘We should not underestimate the online world in which youth live, you know. […] I think that what we just saw, that is mostly taking place on a kind of digital platform, and what we see happening on the streets is the expression of what is happening on the digital platforms.’

Diversity is another crucial aspect in order to stay close to reality. In accordance with the vision of the young adults, it would not be viable to only show people that all look similar regarding ethnic background:

‘Keep it close to reality. I think that is the most important thing. […] That is what you will have to avoid, that you just do not have any diversity you know.’

‘Yeah, well, then you totally have a stigma as well, of course. Suppose you would only choose tan skin-colored boys or only white boys, I think that ultimately evokes the same reaction.’

The professionals agreed with the suggestion to incorporate additional roles into the story, as this would allow for more diversity across the actors. Moreover, increasing the number of people would also be more in accordance with current reality. In addition, the professionals brought in a new perspective with regard to increasing the number of characters. More characters would give room to paint a more nuanced picture regarding the roles of perpetrators and victims. Often, these roles are not so delineated with only a perpetrator on one side and only a victim on the other side:

‘Then you also immediately have the reality of how things are now. Nowadays it is no longer one on one, and you walk alone and someone attacks you. No, it is usually a group. […] Because then it does not matter who is in which group, but then everybody is there. And then, in every group, you have aspects of, well yeah, how they see it, of a loser and a winner. So yes, I think that will come closest to reality, if you would do it the way you have it now.’

Furthermore, the professionals concurred with the decision of switching the roles so you would not have a white person as a victim and someone of color turning to violence, to diminish more structural stigmatization. They recognized that, although ethnicity should not be a relevant factor, it could be a distracting one if you state the obvious stigma. In order to avoid structural stigmatization in the new video, it could be helpful to paint a different picture:

‘I understand that it can be distracting, so, you know, indeed, maybe let Bram [the white boy] be the one throwing a punch, so to speak.’

‘Look, if you turn it into four colored boys, that would of course be even more stigmatizing than the way in which the original video is depicted, so I would absolutely advise against that. And yeah, if you could, indeed, put that white boy in the role of one of those perpetrators, then I think you are there.’

Lastly, the idea behind the escalation in the new scenario, with a more comprehensive story centering around a conflict over a girl including social media, provides a realistic and recognizable storyline according to the professionals:

‘I think it is also very realistic in terms of where youth are now about girls and inciting each other, I think that is very realistic.’ ‘Yeah, yeah, yeah.’

‘Yeah nice, sounds good. I get pretty excited about it when I see it like that. […] So, I think that, with this, you really provide a very realistic context. And I also saw some pictures here and there, and yeah, I think those are all just average adolescents from an average city.’

Overall, the professionals also took into account that the current VR-scenario would be used in a therapeutic, educational context. The scenario would not just be a representation of a possible social situation people might encounter, it would be used with a certain idea behind it and to provide adolescents with an opportunity to develop and practice certain skills. Articulating this idea, and stating that the scenario is not about the way the characters look, for example, also is important to emphasize the factors of interest and avoid too much focus on distracting factors. Nevertheless, of course, that does not mean that distracting factors should not be reconsidered, as currently is being done.

Reflection with screenwriter

Based on the input from the professionals, we summarized the following outcomes: the reality and recognizability of the content are most important, which can be achieved by including social media, diversity and additional characters. Furthermore, avoiding that a white person is depicted as the victim will help to avoid prevalent stigmatization. We concluded that previous decisions adhered to these results.

In addition, taking the previous two co-design groups and our other reflections into account, we decided that most participants thus far explicitly opted for diversity amongst the characters, combined with, in our opinion, the strongest arguments. Therefore, we unequivocally continued with diversity among the characters, as the professionals were again very clear about the necessity of diversity. We elaborated on this by adding examples from the casting agency’s database to all characters in the script. Furthermore, we worked out the storyline regarding social media in more detail and fine-tuned the script so that we could go through it in its entirety during the final co-design group. No additional changes to the draft were made, as no new ideas came forward that needed to be incorporated. For an overview of the third draft and all adjustments that were made through the co-design process, please see the topic list of co-design group 4 (Supplementary material 1).

Co-design group session 4 – 6 adolescents

The last co-design group was fully dedicated to discussing the third draft of the new script, after watching the original video. During the discussion, no fundamental new information came forward, only minor textual adjustments were made. These adjustments regarded particularly slang that was out of date. The adolescents agreed with the storyline of the new script. Especially the importance of a clear reason for the escalation was stressed, in accordance with the first co-design group:

‘They fight very often among themselves. Peer pressure it is, but I don’t know, I have never really experienced someone coming up to me, like, I don’t like you, I am going to beat you up. […] It almost looks like they beat him up because he is white.’

The storyline of the new scenario, in which the conflict centers around a girl and the situation partly escalates through social media as well as that someone starts filming, paints a realistic picture according to the adolescents:

‘I do think it is realistic, it happens so often.’ ‘Yes.’ ‘Guys are so jealous of each other.’ […] ‘I think it is actually perfect, especially that one of those girls starts filming. That is typical.’ ‘Very familiar, very familiar.’ ‘That is so typical.’ ‘Actually, I would do that too, I would just start to film it.’ […] ‘It is actually quite a good fight, because that also happens quite often indeed.’ ‘Yeah.’ ‘Through Instagram.’ ‘Or in real life.’ ‘Or Snapchat, usually those people did not even know each other.’

During this last co-design group, relatively little attention had to be paid to the topic of ethnicity, other than that the adjustments ensured that the characters, their backgrounds and the story around them were now well put together. One of the girls (Moroccan descent) needed a little support from the supervisors to clearly state she thought the original scenario was very stigmatizing and that this stigmatization aligns with her personal experiences as someone of non-Western descent. In addition, these female participants were the first to address the one-sided orientation of the original video in terms of sex and gender without a prompt from the researcher. They emphasized that the topic of the video by no means applies only to boys, highlighting the importance of including female participants.

Reflection with screenwriter

The confirmation of what has been conceived thus far for the new scenario was the main outcome of this last co-design group. Additionally, the textual adjustments and improvements regarding slang were important. Other than that, no fundamental changes were made to the scenario as the adolescents agreed with the last version of the draft and no other modifications were desired. After this, we continued with the definite selection of the actors based on their availability during the shooting day. See Fig. 2 for a picture of the shooting day and Fig. 3 for an image depicting a scene from the new VR-scenario.

Fig. 2

Photo of the shooting day

Fig. 3

Scene from the new VR-scenario

Discussion

By outlining the co-design and development of our enhanced VR-scenario in the current paper, we aimed to provide a clear-cut example of how we took on the moral responsibility that comes with the development of VR-applications. More specifically, we focused on ethnic representation within our virtual environment, to discover how specific bias resulting from ethnic stereotyping could be prevented as much as possible. In addition, our focus was on the participation of important stakeholders and explicit incorporation of reflection during the process of development. By means of this co-design approach, we were able to uncover the thoughts, beliefs and experiences of adolescents, experience experts, and professionals, and gain a deeper and richer understanding of their thoughts, beliefs and experiences [26]. This allowed us to collaboratively make the following most important adjustments to the original scenario: unambiguous inclusion of diversity, keeping the content close to reality, establishing a clear storyline for the scenario, and casting actors in a specific way to avoid obvious stigmatization. As a result, through discussions and reflections on our process and the adjustments that were made, we were able to consolidate the ethical and moral dilemmas surrounding ethnic representation and potentially unethical or immoral behavioral features [10, 18] within our virtual environment.

Results demonstrated that diversity is a key feature that needs to be clearly visible in the content. As put by the experiential experts and professionals, it is important for the content to closely match reality, and reality simply equals diversity. Moreover, the professionals explained that when eliminating ethnic diversity in an attempt to circumvent ethnic bias, as brought in by the adolescents, one potentially introduces an even stronger bias or stereotype. Combining these results with other research, it seems that diversity should also be in line with the reality of social life. A content analysis study of recruitment materials from American colleges and universities showed that, in attempts to portray diversity across their student populations, institutions provided materials that were not accurately representing their student body, as well as showing a narrowed and selective definition of diversity [39]. The portrayed students differed significantly from the actual student body, and students from African American descent were typically portrayed to symbolize diversity as opposed to portraying diversity more representatively. Although such materials presumably come from the best intentions, they might leave people to feel deceived [39]. Thus, although diversity poses to be an important key feature, as stated by most participants in our study, the reality that one wishes to represent in the content must not be forgotten about.

Looking from a more overarching perspective, we can observe there is no unanimously agreed upon solution put forward, as different views and ideas about the best course of action did emerge. Most likely, such individual differences will remain. Not only regarding the perceived best course of action, but also, for example, among people's perception of the extent to which problems with ethnic representation and related matters occur [40, 41]. In our specific case, these differences could relate to the trainers delivering Street Temptations. For instance, in the way they handle situations related to ethnicity and the degree of sensitivity with which they approach these kinds of topics. Would one wish to diminish the influence of these differences to prevent negative experiences, this sensitive topic would explicitly need to be addressed, not only when developing VR-applications, but also when training therapists to work with a VR-application.

Reflecting on the VR-CORE framework we used to guide our study, we believe this framework can be strengthened by incorporating explicit bias assessment in the step of building and testing a prototype. By explicitly taking bias into account in this first phase of VR development, the likelihood of adverse outcomes during subsequent phases of development and validation will decrease. The importance of bias assessment is especially highlighted in a recently published in-depth analysis of potential ethical issues regarding technology use [14]. As stated, merely claiming a technology system does not discriminate against user-groups does not suffice. Specific investigation of potential bias, both with qualitative and quantitative methods, including discussion of the results, is warranted. Although primarily focusing on algorithmic bias, tools for detecting and mitigating bias are available (e.g., [42]). Additionally, the report provides a few checks and balances that apply to VR specifically, which could serve as a starting point to incorporate bias assessment into the phase of prototyping VR-applications.

Strengths and challenges

The iterative co-design process we have described in this paper to operationalize the VR-CORE framework has several strengths. First of all, by means of our study, we do not linger on the question whether or not bias or stereotyping exists. Instead, we started from the assumption that bias is unavoidable, and that it is therefore important to make informed decisions on how to act in a responsible way. Second, by choosing a co-design approach, we explicitly adhered to important process-oriented factors that have been stated in the literature, such as the necessity of direct input from the people at stake [15, 28]. Moreover, co-design allowed us to move beyond just receiving input from the people at stake, striving for a rich and deep understanding of their input and letting this understanding strengthen the collaboratively thought out solutions [26, 27]. Furthermore, we included different stakeholder populations, providing room for feedback from different perspectives. This is important, considering the positive influence of diversity among team members on performance and outcomes [43]. Additionally, the inclusion of different populations has, in combination with the iterative approach, enabled a fresh look at the design throughout the process. All in all, these procedures have helped us to become less prone to harmful bias and to adopt a more sensitive approach.

We must also address some limitations of our co-design study approach. Because we chose to limit the burden on participants by conducting one time co-design groups with each group, we were not able to form a core group of stakeholders that was constantly involved throughout all steps of the process. Therefore, we have not been able to evaluate adjustments with the same people who proposed them, for example. As a result, it might be possible that certain adjustments have been made in a different way than exactly intended by those who brought in the idea. On the other hand, as mentioned above, switching the groups from co-design group to co-design group did allow us to look at the ideas from different angles.

Related to our choice to conduct one time co-design groups was our choice to alternate the co-design groups with separate reflection sessions with the person making the video. We used these sessions to discuss the conclusions and decisions of the different groups and relate them to each other, and to discuss and design additional aspects of the scenario. This approach ensured that the perspectives of different users were included in decisions about the scenario. The decisions made in each of the co-design group meetings were crucial for the next steps in the design process. Each new version was discussed in a new co-design group meeting with other stakeholders. This gave rise to new conclusions and decisions, which were discussed in the next reflection meeting and used as input for the next version of the video. Decisions of stakeholders were, however, not made in mixed group meetings. Thus, our approach did not result in full co-decision. The aspect of co-decision has also been identified as a common challenge in EBCD [33]. Future (experience-based) co-design studies are encouraged to put specific effort in achieving a higher level of co-decision.

Additionally, in light of the forensic youth care context the current study is focusing on, it might have been of added value to work with individuals with lived experience of both the forensic setting and acting. This would follow the example of Teng and Gordon [44], who co-created a VR-based reentry training working together with actors who themselves had a history of imprisonment or who were coached on set by formerly imprisoned people. Individuals with lived experience may act as more credible messengers when trying to convey a certain message [45]. This would also add to the importance of keeping the content close to reality, as stressed by the participants from our co-design groups.

Conclusions

To help circumvent socially irresponsible and conceivably hurtful stereotyping when developing educational or therapeutic VR-applications, having explicit discussions regarding representational bias whilst designing a VR-application is important. Our study shows that it is possible to create VR-content in the context of forensic youth care, circumventing such stereotyping, provided these discussions are based on close collaboration with the people at stake. We suggest applying our approach also in making VR-content in other sensitive areas of mental healthcare. Our paper can both serve as an example of how to shape a co-design process to enable such collaboration, as of an elaboration of specific, informed choices related to ethnic representation in VR. Diversity appears to be an important key feature in this instance, which calls for such informed choices whilst taking different perceptions of individuals into account. Through our co-design process, we consolidated the ethical and moral dilemmas surrounding ethnic representation within our virtual environment. In our opinion, this is an important aspect of developing and working with VR, that should gain an unequivocal role within each developmental process to avoid perpetuation of blind spots in both future research and clinical practice. Accordingly, future research that dedicates itself to the topic of ethnic representation within VR in different therapeutic contexts is highly desirable. We therefore implore other researchers to take on the ethical and moral responsibility that comes with VR development and research.