- Research
- Open access
- Published:
Learner evaluation of an immersive virtual reality mass casualty incident simulator for triage training
BMC Digital Health volume 2, Article number: 56 (2024)
Abstract
Background
To minimize loss of life, modern mass casualty response requires swift identification, efficient triage categorization, and rapid hemorrhage control. Current training methods remain suboptimal. Our objective was to train first responders to triage a mass casualty incident using Virtual Reality (VR) simulation and obtain their impressions of the training’s quality and effectiveness.
We trained subjects in a triage protocol called Sort, Assess, Lifesaving interventions, and Treatment and/or Transport (SALT) Triage then had them respond to a terrorist bombing of a subway station using a fully immersive virtual reality simulation. We gathered learner reactions to their virtual reality experience and post-encounter debriefing with a custom electronic survey. The survey was designed to gather information about participants’ demographics and prior experience, including roles, triage training, and virtual reality experience. We then asked them to evaluate the training and encounter and the system’s potential for training others.
Results
We received 375 completed evaluation surveys from subjects who experienced the virtual reality encounter. Subjects were primarily paramedics, but also included medical learners as well as other emergency medical service (EMS) professionals. Most participants (95%) recommended the experience for other first responders and rated the simulation (95%) and virtual patients (91%) as realistic. Ninety-four percent (94%) of participants rated the virtual reality simulator as “excellent” or “good.” We observed some differences between emergency medical service and medical professionals regarding their prior experience with disaster response training and their opinions on how much the experience contributed to their learning. We observed no differences between subjects with extensive virtual reality experience and those without.
Conclusions
Our virtual reality simulator is an automated, customizable, fully immersive virtual reality system for training and assessing personnel in the proper response to a mass casualty incident. Participants perceived the simulator as an adequate alternative to traditional triage and treatment training and believed that the simulator was realistic and effective for training. Prior experience with virtual reality was not a prerequisite for the use of this system.
Background
Throughout the United States (U.S.), mass casualty incidents (MCIs) are increasing in both number and scope [1, 2]. MCIs may be natural or man-made, although incidents involving shootings or explosions are becoming increasingly common [2,3,4]. As MCIs have increased, there has been a reciprocal focus on training first responders for these types of events [4,5,6]. MCIs, particularly those involving shootings or explosions, require complex decision making since they involve: security threats, life-threatening injuries, challenging triage and transport decisions, coordination among response personnel and interoperability across diverse agencies [7]. Most MCI training efforts to date can be classified into three paradigms: 1) live, large-scale simulations of mass casualty incidents, cast in temporary settings, that involve the use of manikins and actors; [8,9,10,11] 2) tabletop drills that resemble board games; [12] and 3) lectures or presentations on mass casualty response, sometimes followed by facilitated discussions about treating specific patient injuries [13]. These types of simulation platforms provide first responders with methods for increasing knowledge and confidence but are unlikely to provide the sustained practice they need to be able to deliver successful triage and treatment of patients while under the duress of a mass casualty incident [14].
More recently, with the advent of higher-end gaming computers and computer interfaces, virtual reality (VR) systems provide alternative platforms for training individuals in skills required to respond to complex scenarios such as MCIs [15, 16]. Compared to traditional mass casualty incident training methods, VR systems can be programmed to portray the chaotic environment associated with MCIs [17]. Scenarios can be customized to be progressively more challenging through manipulation of noise, debris, and other distractions. Additionally, scenarios can be made more difficult through the addition of more patients who suffer higher degrees of injury acuity. These systems are portable, reusable and allow for continuous practice opportunities for first responders to gradually improve their skills in responding to MCIs [17].
When faced with an MCI, first responders are required to perform two complex tasks. First, they must effectively diagnose the entire scene, which involves sorting and triaging casualty victims into priority groups for transport to hospitals or trauma care centers. Second, they must be able to diagnose and treat those individual patients who are most at risk of losing their life or limb if they are not treated quickly. Both these skills involve clinical reasoning, which is best learned through repetitive, deliberate practice with performance feedback from an instructor [18].
One triage protocol that has been shown to be effective is the Sort, Assess, Lifesaving interventions, and Treatment and/or Transport (SALT) Protocol [19]. This protocol was proposed by a Center for Disease Control (CDC) sponsored working group to standardize triage methods across the U.S. The SALT Protocol is evidence-informed and endorsed by numerous professional organizations including: the American College of Emergency Physicians, American College of Surgeons Committee on Trauma, National Association of EMS Physicians, and National Disaster Life Support Education Consortium [19]. SALT has several advantages over other protocols. First, it can be applied to both pediatric and adult casualties. Second, it saves time by using voice commands to perform an initial global sort. Third, it eliminates the requirement of having to count respirations or check capillary refill, which are challenging tasks in chaotic environments. Finally, SALT allows for more rapid application of life saving interventions (LSIs) such as tourniquets [13].
The primary purpose of this project was to evaluate the use of a VR platform for training first responders in the skills required for disaster response, specifically skills related to SALT triage and treatment of an MCI. We were guided in our design of this VR simulation platform by the foundational elements of Brain-based Learning Theory, which includes the principle that learning complex clinical tasks is maximized when learners can experience immersion in realistic settings [20, 21]. Because little is known about whether learners would find the VR platform to be an acceptable alternative to traditional disaster training platforms, our objective was to gauge learners’ reactions to this fully immersive VR experience with triaging and treating patients in response to a virtual MCI. A secondary purpose was to determine whether prior experience using VR systems was associated with the participant’s perceptions about the quality of the VR training encounter.
Methods
Study design, setting and participants
This study was a prospective observational study of learner impressions of a fully automated, fully immersive VR system for training and assessing personnel to effectively respond to an MCI [17]. The system uses a commercially available laptop gaming computer and a Meta Quest 2 VR headset, (Reality Labs, Meta Platforms, Menlo Park, CA. https://www.meta.com/quest/products/quest-2/) and was designed for portability and practical use by agencies responsible for training first responders. We used the system as part of a short program to train learners in implementing the SALT Triage protocol [22].
Training was delivered to Emergency Medical Service (EMS) clinicians: paramedics and EMTs, from agencies throughout Central Ohio. EMS training sessions were held at the agency’s training facilities or in their service bays. We also held SALT Triage training sessions for future and current emergency medicine physicians: medical students, emergency medicine (EM) residents, and EM physicians. Physicians who are boarded in emergency medicine are commonly involved in the training of EMS professionals in disaster medicine skills. Medical learner training was conducted in medical school classrooms. Study participants completed consent forms and were screened for susceptibility to motion sickness (e.g. vertigo, dizziness), a potential side effect of VR. This study was approved by The Ohio State University Institutional Review Board (Columbus, OH, USA).
SALT triage training
The SALT Triage training was delivered in four parts. First, learners were introduced to the protocol through a brief lecture. Most learners already knew how to deliver life-saving treatments such as application of a tourniquet, so for those groups, we focused attention on the global sort technique and the definitions of each level of triage. More novice learners were also taught how to perform life-saving treatments. Second, each learner entered the VR simulator and participated in an orientation of the system. Third, the learners were provided the opportunity to perform SALT Triage on a simulation scenario that contained eleven patients, all of whom required triage categorization, and more than half required life-saving treatments. Finally, after the learner’s VR encounter, they received feedback through a debriefing session that was guided by a printed report of their performance. During debriefing, learners were offered the opportunity to explain their reasoning behind the decisions they made during the encounter.
MCI-VR simulator
The VR simulator scenario involved the recreation of a terrorist bombing of an underground subway station (Fig. 1). The system is fully described elsewhere but will be briefly summarized here [17]. Participants entered the MCI scene by donning a VR Head-Mounted Display (HMD) and hand-held controllers. While participants could physically walk around the virtual space, the controllers made navigation more efficient through the “teleport” (point to the spot they want to go and click) or “strafe/glide” (small movement) features, which moved them to where they wanted to go without walking. This means that the physical space for the encounter did not have to be the full size of the virtual subway platform, which if converted to actual physical space would be approximately 60 by 30 feet. For safety, participants were provided with a guardian (a research staff member) to ensure that they did not physically venture into physical obstacles.
In VR, participants were equipped with a virtual medical kit. They used the controllers to grab tools and triage tags from the kit, or to take a pulse on a patient. Virtual patient injuries were those commonly suffered by individuals exposed to an explosion. Subsequently, learners could implement various lifesaving interventions (LSIs) including opening an airway of someone struggling to breathe, controlling major hemorrhage with a tourniquet or wound packing, or needle decompressing the chest of a victim who is suffering from a tension pneumothorax. Following completion of the encounter, the participants removed the headset and moved on to the debriefing session.
The MCI-VR System recorded and tabulated essentially every action the learner performed during the simulation encounter to produce a performance report. Faculty evaluators used this performance report to provide feedback to the learner through a post encounter debriefing on their performance.
Measures
Post-encounter surveys were designed by the researchers primarily to assess learner satisfaction with their VR experience. In addition to items that evaluated features of the system, we asked participants for an overall judgement of its training potential and whether the system encounter supported their learning needs. To gauge whether participants needed prior VR experience to benefit from the encounter, we used ownership of a VR system as a proxy for experience. We asked two open-ended questions about the system’s strengths (effective features of the system), and things we might do to improve it. Members of the VR development team used feedback from the second open-ended item to fine tune the MCI-VR system during Beta testing. The survey also included requests for personal characteristics such as: professional role, level of training, and prior triage experience (Appendix A).
Analysis
We described our participant characteristics using frequencies and percentages and compared professional groups (EMS vs. Medicine) using Chi Squares (Fisher’s Exact Tests). To assess whether participants’ evaluations of the MCI-VR system features were associated with prior VR experience, we compared those who owned a VR system, to those who did not, using independent t-tests. For items related to how well the MCI-VR encounter contributed to the participants’ learning, and opinions about the utility of the MCI-VR system, we compared EMS to medical professionals using independent t-tests. For statistically significant results, we calculated associated effect sizes: either Cohen’s D for mean statistics or Cramer’s V for Chi Squares. Statistical analyses were performed with IBM-SPSS, Version 29.0.1 (IBM Corp. Released 2022. IBM SPSS Statistics for Windows, Version 29.0.1 Armonk, NY: IBM Corp). To control rates of Type-1 error, we used the Bonferroni method to calculate family-wise levels of significance by dividing the desired P = 0.05 level by the number of comparisons within in each set (k/0.05) [23].
We used summative content analysis to code and reduce open-ended survey responses to themes [24,25,26]. First, one author (ARP) drafted a preliminary coding dictionary based on anticipated survey responses and common responses received in program evaluations. Second, responses to open-ended questions were entered into an Excel workbook (Excel, Version 2211, Redmond, WA, USA). Third, other authors (ARP, DPW, NEK) used word units (i.e., words, word stems, and words phrases with common semantic meanings) to assign codes to each response from the coding dictionary. Comments about strengths and suggestions for improving the VR system were coded separately. Coding assignments were performed iteratively as: 1) the coding dictionary was refined, 2) codes that shared similar underlying concepts were combined, and 3) codes that contained fewer than ten responses were eliminated. Excel continued to be used to process responses into codes until the dataset was effectively reduced to the strongest themes based on the number of times responses appeared.
Results
Between March 2022 and May 2023 we trained a total of 536 participants using our SALT Training Program which included encounters with the MCI-VR simulation system. Of the individuals trained, 83% (447 of 536) completed post-encounter surveys, however, we excluded 73 individuals who completed less than half of the survey (13.6%; 73 of 536). Subsequently, our final post-encounter survey response was 70% (375 of 536).
Participant characteristics
Eighty-six percent (86%, 324 of 375) of respondents were EMS professionals (Basic EMTs or Paramedics), and 61% (230 of 374) of respondents considered themselves to be seasoned first responders. One quarter of our subjects (25%, 92 of 374) indicated enthusiasm for computer games, while 17% (64 of 374) reported owning a VR system (Table 1).
To gauge respondents prior triage training and experience, we asked them to identify specific courses they had taken, and the number of large-scale disaster drills in which they had participated. Seventy percent (70%, 262 of 374) of respondents had completed some form of triage training, while only 33% (124 of 374) had been trained specifically in SALT Triage. Fifty-five percent (55%, 206 of 374) had completed a formal disaster response training program; and of that group, the mean number of disaster drills in which they participated was 2.3 drills (Std. Dev. = 0.90) (Table 1).
EMS professionals differed significantly from medical learners regarding training and professional identity as a first responder. Significantly more EMS professionals than medical learners considered themselves to be “seasoned first responders” (P ≤ 0.001, es = 0.44). Participants who identified as seasoned first responders were also significantly more likely to have: completed some form of triage training other than SALT (P ≤ 0.001, es = 0.29); completed a formal disaster response training course (P ≤ 0.001, es = 0.29); and participated in significantly more large-scale disaster drills than those who did not think they were seasoned first responders (P ≤ 0.04, es = 0.22). The effect sizes associated with these differences are considered intermediate in size (Table 1).
Evaluation of VR features
Nearly all participants indicated that the VR orientation helped them to master the use of controllers to navigate the virtual space, (96%, 359 of 374) and subsequently felt sufficiently prepared to enter the MCI-VR subway station scenario (93%, 348 of 374). Almost all participants identified the MCI-VR simulator (95%, 354 of 374) and the patients within as realistic (91%, 340 of 374). They also reported that patients responded to their commands during the encounter (85%, 319 of 374). Participants agreed that the medical kit contained all the tools they needed to field treat patients (91%, 317 of 374), and that tools were easy to use (90%, 335 of 374). We observed no differences in evaluation responses from participants who own VR equipment and those who do not (Table 2).
Despite the large numbers of participants who found the orientation helpful, 18% percent (66 of 374) of subjects said that VR navigation (a skill they learn during orientation) was challenging. Furthermore, 26% (98 of 374) believed they needed more time to acclimate to the VR environment before engaging the MCI-VR subway station (Table 2).
Contribution to learning
While the instructional program designed for use with the MCI-VR system was not specifically intended to be a summative performance assessment, we asked participants about their views regarding the accuracy of the system generated score and whether their encounter performance was a valid assessment of their first responder skills. Both EMS and medical professionals responded similarly to these items, with about three-quarters of participants saying that the score was accurate (77%, 288 of 374) and that it was a valid assessment of their skills (74%, 276 of 374) (Table 3). However, EMS professionals rated the value of feedback and practice with the system significantly lower for making them more effective as a first responder (P ≤ 0.001, es = 0.55). The Cohen’s D effect sizes for these differences in mean ratings were considered intermediate in size.
Future potential of MCI-VR for training first responders
Participant’s general response to the MCI-VR Simulation encounter was positive, with nearly all stating that they would recommend this experience to other first responders (96%, 360 of 374) and assigning it a letter grade of “A” for excellent or “B” for good. (93%, 349 of 374) (Table 4). Despite these positive ratings, only 74% (277 of 374) viewed the MCI-VR training as effective as live training.
Summative content analysis
Three-quarters of participants provided comments about strengths of the MCI-VR System (75%, 279 of 347), while fewer than half (49%, 184 of 347) provided suggestions for improving the program.
Strengths of the MCI-VR system
Five key themes about strengths of the MCI-VR program were identified by the content analysis: realism, interactivity, and the opportunities to practice triage and treatment skills, decision making, and managing multiple patients (Table 5). The most common comment was about the realism of the simulation, some specifically about the realism of the virtual patients: “I thought the patient responses/panic even when they were “minimal” was a realistic portrayal of the emotional influence that patients have on providers despite the severity of injuries.” Several participants recognized the value of the MCI-VR system, including some who appreciated the opportunity to practice without having to risk harm to “real” patients: “Able to make mistakes without hurting real people.” A number of participants mentioned the interactivity of the MCI-VR system and the patients within as being key to learning and practicing triage and treatment skills: “The physical aspect of interacting with patients, treating and assigning them tags was really helpful,” “This was a good refresher and helped remember what equipment is needed.” Participants also remarked about the importance of experience with triage decision making. Specifically, the simulation pushed participants to “manage an overwhelming situation and critically think while staying focused.” Finally, several participants recognized that one key advantage of the VR simulation experience was the opportunity to manage multiple patients collectively instead of one at a time like in tabletop exercises: “I enjoyed having to manage multiple patients in rapid sequence. This gave me the ability to go from one patient to the next. Other trainings like this present one standardized patient at a time and those patients may not be trained to act or behave appropriately when injured in that manner.”
Suggestions for improving the MCI-VR system
Most suggestions for improvement had to do with modifying the system by either improving or adding features (Table 6). Many suggestions, particularly those involving patient interactivity, or injury visibility, were used by programmers to make improvements to the system over the period in which these evaluations were conducted. Some features requested by participants were already built into the system but were not yet made available during their one-time encounter. These include the ability to increase the difficulty of the scenario by increasing the levels of chaos: “Building on this to add more stressors or distractions would be great,” and “Make patient status dynamic during the exercise.” Other suggestions involved possible features for the future, such as adding tools to the medical kit (chest seals, trauma sheers, or a stethoscope), or building virtual sets that resemble environments within the participant’s jurisdiction. Some participants recognized the value in being able to have more than one participant in the MCI-VR environment at the same time: “Make it multiplayer so crews can work in concert.” A substantial number of participants experienced system glitches, such as the system freezing or the VR headset losing connectivity to the laptop computer on which the program runs. Glitches occurred regularly at the beginning of Beta testing but subsided over time. Comments about glitches also declined as problems were fixed. Finally, there were a few participants who felt the need for more time in the orientation to become more adept at using the controllers: “I needed a little more time familiarizing myself with the controls”.
Discussion
Participants of the SALT Training with the MCI-VR System would recommend this experience to others and graded the experience as an “A” for excellent or “B” for good. The system was equally received as an effective training experience by both prehospital and hospital care providers, regardless of prior experience with triage or disaster training. Both groups credited the feedback from the activity for improving their first responder skills and believed that continued practice with the system would make them more effective first responders. While some participants commented on the need for more time to acclimate to the VR environment, we found no statistical differences in ratings of the system between those who owned VR systems and those who did not. Participants rated the system’s realism and interactivity highly, and reinforced these ratings with similar comments about how these were strengths of the system.
Three-quarters of our participants considered the MCI-VR training to be as effective as live, large-scale exercises, which may be more realistic, but are expensive, resource intense, requiring substantial planning and amounts of space. This finding suggests that VR for disaster response training is a viable option and could provide EMS agencies more flexibility in offering their first responders opportunities to learn and practice triage and treatment skills in realistic recreations of mass casualty incidents.
Virtual reality technology has gradually improved over the past ten years to become more immersive and realistic. Earlier versions of VR technology for training disaster response skills, such as Second Life, resembled computer versions of tabletop drills by presenting content on 2D computer screens. Participants interacted with these versions by controlling avatars through keyboards and mouse controls [29, 30]. VR became more immersive with the use of computer automated virtual environments (CAVEs), which used wall projection and liquid crystal glasses to create simulated disaster environments [31, 32]. CAVEs immersed subjects in the simulation rather than watching an avatar on a computer screen. The major drawback to these CAVE systems was that they employed expensive technology that limited their use to institutions with simulation centers and large simulation budgets [33].
Recent advances in VR technology now allow for the creation of realistic, highly immersive experiences at a much lower cost than these earlier systems. Our simulation used a MetaQuest 2 VR head-mounted display with controllers that allowed for individual tracking, tactile pulse checks, and highly immersive simulations. As such, VR simulations that were previously restricted to very expensive systems in large simulation centers can now be created at significantly less cost and be made accessible to a wider variety of organizations from university or medical school training programs to local fire, rescue, and EMS agencies. Moreover, assessments can be automated and built into the simulation, further increasing the utility of newer VR approaches.
Our VR simulation illustrates the practicality of this technology for disaster and mass casualty training. The MCI-VR system was well received by a variety of learners at different levels of training and did not rely on prior computer gaming experience, nor prior experience with disaster training or drills. Literature on optimal ways to prepare learners for VR environments was limited, so we designed our own brief orientation to help learners transition. The responses of our learners made clear that for most, a brief (approximately 10 min) guided tutorial and orientation was an effective way to prepare them for the actual simulation. This tutorial allowed the most novice VR learners to focus on the triage and treatment of patients without having to worry about the “knobology” of controlling the VR system.
One aspect of the training praised by learners was the value of the expert debriefing. Although the system generated a score report that could be used by the learner, we chose to provide formative feedback by reviewing these reports with participants in person. Our customizable system provided a structure that could leverage simulation-based mastery learning to gradually improve a learner’s mastery of triage and treatment skills over time [34]. Furthermore, this system provides a living laboratory for studying and evaluating triage protocol efficacy and the performance of learners from various backgrounds.
Limitations
This project examined participants’ evaluation of SALT triage training using the MCI-VR system, which is analogous to their satisfaction with the experience. Subsequently, we are not able to conclude that the training improved participant triage and treatment performance. Nor are we able to conclude that patient outcomes would improve because of improved first responder skills. Future research will be needed to attain these higher-level objectives. Although we screened for motion sickness prior to entry into the MCI-VR system, we did not ask participants about the impact of the system on them, either physiologically or psychologically, after the encounter [35]. While we assume from their positive evaluations that participants responded to the simulation similarly to a real-life incident, we were not able to verify this assumption.
Future research
Evaluation feedback has been used to improve the MCI-VR simulator including the expansion of the tools available in the medical kit and responsiveness of the virtual patients. Respondents have encouraged further development of the system, suggesting the inclusion of other disaster scenarios, and increasing the capacity of participants in the system for developing teamwork and communication. We are currently analyzing system generated data to investigate participant’s performance: such as time to hemorrhage control, and accuracy and efficiency in triaging and treating the scene. Further, we are planning subgroup analyses to compare the performances of first responders from varied training backgrounds (medical students, residents, paramedics and EMTs). Finally, we plan to leverage the VR system for studying human factors such as stress and their relationship to first responder performance.
Conclusions
Our VR simulator is an automated, customizable, fully immersive virtual reality system for training and assessing personnel in the proper response to a mass casualty incident. Participants perceived the encounter as realistic and effective for training, regardless of their prior level of training or their experience using virtual reality. The MCI-VR simulator was demonstrated to be a viable training alternative to current methods of training disaster response.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author upon reasonable request.
Abbreviations
- VR:
-
Virtual Reality
- SALT:
-
Sort, Assess, Lifesaving interventions, and Treatment and/or Transport
- EMS:
-
Emergency Medical Service
- U.S.:
-
United States
- MCI:
-
Mass Casualty Incident
- CDC:
-
Center for Disease Control
- EM:
-
Emergency Medicine
- HMD:
-
Head mounted display
- LSI:
-
Life saving intervention
- MCI-VR:
-
Mass casualty incident virtual reality
- P:
-
Probability under the null hypothesis
- K:
-
Number of pair-wise comparisons
- Es:
-
Effect size
- EMT:
-
Emergency Medical Technician
- N:
-
Number of subjects
- Mn:
-
Mean
- SD:
-
Standard Deviation
- D/A:
-
Disagree/Agree Equally
- AI:
-
Artificial Intelligence
References
National Crime Victims’ Rights Week Resource Guide: Crime and Victimization Fact Sheets: Mass Casualty Shootings. Office for Victims of Crime, U.S. Department of Justice, Office of Justice Programs; Washington DC: 2018. https://ovc.ojp.gov/sites/g/files/xyckuh226/files/ncvrw2018/info_flyers/fact_sheets/2018NCVRW_MassCasualty_508_QC.pdf. Accessed 19 Jan 2021.
Peterson J, Densley J. The Violence Project: How to stop a mass shooting epidemic. New York, NY: Abrams Press; 2021.
Krouse WJ, Richardson DJ. Mass Murder with Firearms: Incidents and Victims, 1999–2013. Congressional Research Service Report. Washington DC; Library of Congress: 2015. http://www.fitgny.com/uploads/7/5/7/0/75709513/congressional_research_paper_mass_shootings.pdf. Accessed 21 Jan 2021.
Schwartz RB, McNut, R. The Explosive Mass Casualty Incident: Prehospital Incident Management and Triage. In: Callaway, D., Burstein, J. (eds) Operational and Medical Management of Explosive and Blast Incidents. Cham, Switzerland: Springer Nature; 2020. p. 189–91. https://doi.org/10.1007/978-3-030-40655-4_13.
Biswas S, Bahouth H, Solomonov E, Waksman I, Halberthal M, Bala M. Preparedness for mass casualty incidents: the effectiveness of current training model. Disaster Med Public Health Prep. 2022;16(5):2120–8. https://doi.org/10.1017/dmp.2021.264.
Frykberg ER. American College of Surgeons: Statement on disaster and mass casualty management. Bull Am Coll Surg. 2003;88(8):14–5.
Mitchell E. Fire/emergency medical services department operational considerations and guide for active shooter and mass casualty incidents. United States Federal Emergency Management Agency, United States Fire Administration, Washington, DC. 2013. https://www.usfa.fema.gov/downloads/pdf/publications/active_shooter_guide.pdf. Accessed 1 July 2023.
Roccaforte JD, Cushman JG. Disaster preparedness, triage, and surge capacity for hospital definitive care areas: optimizing outcomes when demands exceed resources. Anesthesiol Clin. 2007;25(1):161–77. https://doi.org/10.1016/j.anclin.2007.01.002.
TariVerdi M, Miller-Hooks E, Kirsch T. Strategies for improved hospital response to mass casualty incidents. Disaster Med Public Health Prep. 2018;12(6):778–90. https://doi.org/10.1017/dmp.2018.4.
Kman NE, Bachmann D, Folley AT, Adams J, Greer M. Emergency preparedness simulation cases for medical students: crush and organophosphate exposure. MedEdPORTAL. 2013;9:9330. https://doi.org/10.15766/mep_2374-8265.9330.
Farra SL, Gneuhs M, Hodgson E, Kawosa B, Miller ET, Simon A, Timm N, Hausfeld J. Comparative cost of virtual reality training and live exercises for training hospital workers for evacuation. Comput Inform Nurse. 2019;37(9):446–54. https://doi.org/10.1097/2FCIN.0000000000000540.
McGlynn N, Claudius I, Kaji AH, Fisher EH, Shaban A, Cicero MX, Santillanes G, Gausche-Hill M, Chang TP, Donofrio-Odmann JJ. Tabletop application of SALT triage to 10, 100, and 1000 pediatric victims. Prehosp Disaster Med. 2020;35(2):165–9. https://doi.org/10.1017/S1049023X20000163.
Deluhery MR, Lerner EB, Pirrallo RG, Schwartz RB. Paramedic accuracy using SALT triage after a brief initial training. Prehosp Emerg Care. 2011;15(4):526–32. https://doi.org/10.3109/10903127.2011.569852.
Baetzner AS, Wespi R, Hill Y, Gyllencreutz L, Sauter TC, Saveman BI, Mohr S, Regal G, Wrzus C, Frenkel MO. Preparing medical first responders for crises: a systematic literature review of disaster training programs and their effectiveness. Scand J Trauma Resusc Emerg Med. 2022;30(1):76. https://doi.org/10.1186/s13049-022-01056-8.
Iyer AV, Dunlop SR, Thakkar DJ, Mishra S, Banerjee A, Gokhale A, Shah K, Faiz A, Arora A, Awasthi P, Gebbie M. Evaluation of Current Technologies for Training, Web Apps, and New Technologies. Joint Transportation Research Program Publication No. FHWA/IN/JTRP-2021/15. West Lafayette, IN; Purdue University: 2021. https://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=3295&context=jtrp. Accessed: 22 Mar 2022.
Allcoat D, von Mühlenen A. Learning in virtual reality: Effects on performance, emotion and engagement. Res Learn Tech. 2018;26:2140. https://doi.org/10.25304/rlt.v26.2140.
Kman NE, Price A, Berezina-Blackburn V, Patterson J, Maicher K, Way DP, McGrath J, Panchal AR, Luu K, Oliszewski A, Swearingen S, Danforth D. First Responder: Virtual Reality Simulator to Train and Assess Emergency Personnel for Mass Casualty Response. JACEP Open. 2023;4(1):e12903. https://doi.org/10.1002/emp2.12903.
Ericsson KA. Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice. Acad Med. 2015;90(11):1471–86. https://doi.org/10.1097/ACM.0000000000000939.
Lerner EB, Schwartz RB, Coule PL, Weinstein ES, Cone DC, Hunt RC, Sasser SM, Liu JM, Nudell NG, Wedmore IS, Hammond J. Mass casualty triage: an evaluation of the data and development of a proposed national guideline. Disaster Med Public Health Prep. 2008;2(S1):S25-34. https://doi.org/10.1097/DMP.0b013e318182194e.
Rodgers DL. High-fidelity patient simulation: a descriptive white paper report. Healthc Simul Strategies. 2007;10(4):68–77. Available: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=d882b2d4e333af26d0e6e3412bf2644a53bbac15.
Caine RN, Caine G. Making connections: Teaching and the human brain. Alexandria, VA; Association for Supervision and Curriculum Development: 1991. ERIC ED 335–141. https://files.eric.ed.gov/fulltext/ED335141.pdf. Accessed: 22 Apr 2019.
Kman NE. Salt Mass Casualty Triage, Just in Time SALT Triage Video. Dublin, OH; Columbus Medical Association, Central Ohio Trauma System: 2023. https://www.cotshealth.org/salt. Accessed: 1 Sept 2023.
Bland JM, Altman DG. Multiple significance tests: the Bonferroni method. BMJ. 1995;310(6973):170. https://doi.org/10.1136/bmj.310.6973.170.
Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88. https://doi.org/10.1177/1049732305276687.
Ayres L. Thematic Coding and Analysis. In: Given LM, editor. The Sage encyclopedia of qualitative research methods. Thousand Oaks, CA: Sage publications; 2008. p. 867–8.
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. https://doi.org/10.1191/1478088706qp063oa.
Hahs-Vaughn DL, Lomax RG. An introduction to statistical concepts. Philadelphia, PA: Routledge; 2020. p. 312.
Lenhard W, Lenhard A. Computation of effect sizes. Psychometrica. 2022. Retrieved from: https://www.psychometrica.de/effect_size.html. https://doi.org/10.13140/RG.2.2.17823.92329.
McGrath J, Kman N, Danforth D, Bahner DP, Khandelwal S, Martin DR, Nagel R, Verbeck N, Way DP, Nelson R. Virtual alternative to the oral examination for emergency medicine residents. West J Emerg Med. 2015;16(2):336–43. https://doi.org/10.5811/westjem.2015.1.24344.
Cone DC, Serra J, Kurland L. Comparison of the SALT and Smart triage systems using a virtual reality simulator with paramedic students. Eur J Emerg Med. 2011;18(6):314–21. https://doi.org/10.1097/MEJ.0b013e328345d6fd.
Andreatta PB, Maslowski E, Petty S, Shim W, Marsh M, Hall T, Stern S, Frankel J. Virtual reality triage training provides a viable solution for disaster-preparedness. Acad Emerg Med. 2010;17(8):870–6. https://doi.org/10.1111/j.1553-2712.2010.00728.x.
Wilkerson W, Avstreih D, Gruppen L, Beier KP, Woolliscroft J. Using immersive simulation for training first responders for mass casualty incidents. Acad Emerg Med. 2008;15(11):1152–9. https://doi.org/10.1111/j.1553-2712.2008.00223.x.
Vincent DS, Sherstyuk A, Burgess L, Connolly KK. Teaching mass casualty triage skills using immersive three-dimensional virtual reality. Acad Emerg Med. 2008;15(11):1160–5. https://doi.org/10.1111/j.1553-2712.2008.00191.x.
Sawyer T, White M, Zaveri P, Chang T, Ades A, French H, Anderson J, Auerbach M, Johnston L, Kessler D. Learn, see, practice, prove, do, maintain: an evidence-based pedagogical framework for procedural skill training in medicine. Acad Med. 2015;90(8):1025–33. https://doi.org/10.1097/ACM.0000000000000734.
Tovar MA, Zebley JA, Higgins M, Herur-Raman A, Zwemer CH, Pierce AZ, Ranniger C, Sarani B, Phillips JP. Exposure to a Virtual Reality Mass-Casualty Simulation Elicits a Differential Sympathetic Response in Medical Trainees and Attending Physicians. Prehosp Disaster Med. 2023;38(1):48–56. https://doi.org/10.1017/S1049023X22002448.
Acknowledgements
The authors would like to acknowledge the learners who provided evaluation feedback on their training experience and the virtual reality training platform. These include EMS Professionals from the Central Ohio Region, and medical students and emergency medicine residents from The Ohio State University. Participant feedback was invaluable at fine-tuning the program for future learners.
Authors' information
ARP, NEK and JM are professors of emergency medicine and emergency medicine physicians. ARP specializes in research, training and development of EMS Professionals and is the Research Director at the National Registry for EMTs. NEK specializes in disaster medicine and is a member of Ohio Task Force 1, Disaster Response Team. JM specializes in alternative methods of performance assessment for emergency medicine residents. DD is a professor of Obstetrics and Gynecology and specializes in virtual reality development, including the development of virtual patients. DPW is an educational research scientist who specializes in educational research, program evaluation, measurement, and performance assessment. AP VBB, JP are specialists in computer graphics, interactive virtual reality program and development, and computer arts and design. JP is also a specialist in database engineering.
Funding
This project was funded under grant number GR126415 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services (HHS). The authors are solely responsible for this document’s contents, findings, and conclusions, which do not necessarily represent the views of AHRQ. Readers should not interpret any statement in this report as an official position of AHRQ or of HHS. None of the authors has any affiliation or financial involvement that conflicts with the material presented in this report.
Author information
Authors and Affiliations
Contributions
ARP, DPW, AP, VBB, JP, JM, DD, NEK contributed to the development of the virtual reality program used for this study. NEK, ARP and JM contributed to the development of the teaching materials related to the SALT Training program. DPW, ARP and NEK contributed to drafting the original manuscript. All authors contributed to writing and revisions of the manuscript, and all authors read and approved of the final manuscript submission.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The study was approved by the Ohio State University Institutional Review Board. Participants in this study completed informed consents for their deidentified data (evaluation forms) to be used for this research.
Consent for publication
Written informed consent for publication has been obtained from the participants in this study. Please note that patients portrayed in Fig. 1 are artist renderings and are not real or live patients.
Competing interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Way, D.P., Panchal, A.R., Price, A. et al. Learner evaluation of an immersive virtual reality mass casualty incident simulator for triage training. BMC Digit Health 2, 56 (2024). https://doi.org/10.1186/s44247-024-00117-5
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s44247-024-00117-5