Immersive Learning with Virtual Field Visits: Spherical Video-Based Virtual Reality of Factory Environments

AN0177084372;[2wzs]01may.24;2024Nov14.10:02;v2.2.500

Immersive Learning with Virtual Field Visits: Spherical Video-Based Virtual Reality of Factory Environments 

The authors develop, use, and evaluate a spherical video-based virtual reality (SVVR) app for teaching operations management. The app contains a 360-degree photo-realistic copy of real factory environments with blended information and gamified exercises. To solve a learning assignment, students virtually immerse themselves in the factory using the app on their smartphones with virtual reality headsets, tablets, or computers. The authors developed the SVVR app together with an industrial partner, implemented it in a graduate-level course on production and operations management, and evaluated it by drawing on app user data, assignment grade scores, and an evaluation survey. The results show that students reported an overall positive learning experience and that students who engaged more in the SVVR app performed better on the assignment. This paper contributes to the literature on SVVR with evidence of the learning effects of SVVR in an operations management curriculum. It provides an example of how SVVR enables virtual factory visits, integrates gamification elements, and can be integrated into teaching. Funding: This work was supported by the ETH Zurich Innovedum.

Keywords: virtual reality; mixed reality; digital factory; teaching

1. Introduction

Field trips to companies have been an integrated part of many industrial engineering and management courses because they provide a first-hand, immersive learning experience. Field trips help students connect abstract concepts and principles to real-life applications ([15], [39]). In recent years, information technologies have been proposed to fully or partially replace physical field trips ([40], [44], [33], [24]). An up-and-coming technology for virtual field trips is spherical video-based virtual reality (SVVR). This technology situates students in a photo-realistic copy of an environment without the need for travel.

Enabled by rapid technology development, SVVR has recently been introduced in the teaching of a range of disciplines, including language education ([21]), geography ([17]), earth science and geology ([7]), chemistry and biology ([19]), medical studies ([43]), and therapy ([4]), among others. Reviewing the literature on immersive virtual reality-based teaching, [36] identified 18 application areas across 38 papers, with engineering and computer science as the most frequently reported disciplines. They find that most studies focus on high-end virtual reality technologies, like Oculus or HTC VIVE, and less than one-fifth of the studies on the use of low-end immersive virtual reality. While this bias is reasonable from a technology development perspective, low-end SVVR has much higher adoption and scalability potential due to its lower cost and widespread availability of consumer technologies like smartphones, computers, and 360-degree cameras. More importantly, few of the reviewed studies report and evaluate actual implementation in teaching ([36]). This paper addresses these shortcomings in the literature by reporting the development, use, and evaluation of a low-end SVVR in a new setting: operations management.

The discipline of operations management teaches students the principles of using industrial processes to transform inputs into outputs for a market ([1], [2]). This body of knowledge is also taught in courses on industrial engineering, industrial management, production management, and the like. The main activities covered include manufacturing processes in factories and shipment to logistics centers and customers. To enhance the learning experience, a common teaching practice is to take engineering and business students on company visits with immersive factory tours. Such field trips are challenging to organize, particularly for large classes. They take students off campus for several hours, which may interfere with other classes, and are limited by companies' capabilities and willingness to host a large number of students ([15]). It has therefore been proposed that such field trips can effectively be replaced with virtual field trips ([26]). However, different from, for example, geology courses that can use freely captured 360-degree photos or videos in nature ([7]) or therapy courses that can use acted role-play, the best way to provide a realistic and fully immersive experience in operations management requires the recording of a real factory or logistics center. This is a great challenge due to intellectual property rights and perhaps the main reason why SVVR has found minimal use in operations management to date. This paper aims to demonstrate and empirically evaluate an example of how SVVR can be used to improve the learning experience and learning outcomes in operations management and related courses. Besides presenting the SVVR app used in a course, this paper asks, what are the effects of such low-cost SVVR on students' (1) learning experience and (2) learning outcomes?

The authors created an SVVR app called FactoryVR containing a 360-degree virtual copy of a manufacturer's assembly factory. The photo-realistic immersive virtual environment was blended with information and gamified content. To solve a graded learning assignment as part of the course Production and Operations Management at ETH Zurich, students had to immerse themselves in the SVVR material. The 2020 course was attended by 170 students who performed the SVVR-based assignment in groups of four (two groups contained three students). Because of COVID-19 restrictions, the students could only meet virtually, and there was no way a physical factory visit could be organized for the students. The authors evaluated the effects on learning outcomes by regressing app user statistics on assignment grading points. In addition, an evaluation survey and course feedback forms were distributed to gain insights into the students' learning experiences. Considering the positive reports, the FactoryVR app has been adopted in the standard course curriculum.

The rest of the paper is organized as follows. Section 2 reviews the literature on virtual reality in teaching and its pedagogical foundation. Section 3 presents how the FactoryVR app was created and what it contains. Section 4 outlines the details of how the app was integrated into teaching. Section 5 reports three different analyses of the relationship between the SVVR assignment and students' learning outcomes and learning experiences. Section 6 elaborates on the implications of the results and experiences for learning and teaching operations management. In Section 7, a short conclusion is provided.

2. Background

Virtual reality has a long history. Its start is often credited to science fiction author Stanley G. Weinbaum's Pygmalion's Spectacles from the 1930s ([46]) or Ivan [42] article The Ultimate Display ([42]). Although initially developed for entertainment and military use, virtual reality has great potential in training and education ([34]) and is subject to a rapidly expanding literature ([36], [47], [14], [23], [32]). Because of technological advancement, virtual reality started to be adopted more broadly around mid-2010.[1] It comprises a range of computer technologies, from simple three-dimensional videos ([38]) to fully immersive simulators and real-time, but online, classrooms and meetings ([16]).

The most striking feature of virtual reality is that it enables telepresence—or virtual presence—defined as the feeling of being physically present in a different environment than the one where the user is located ([37], [8]). The virtual environment can be a copy of, similar to, or entirely different from the real world. When the virtual environment is a photo-realistic copy of the real world—captured by 360-degree photos or videos—the literature speaks of "SVVR." SVVR is different from computer-generated virtual environments because it ensures an immersive experience that is closer to what one would experience in the real world if going there physically. This is a great pedagogical advantage for courses that teach real-life content such as biology, geology, architecture, or management. Accordingly, a growing literature is studying the use and effects of SVVR in teaching.

The literature on the pedagogy of virtual reality in teaching suggests that it can be an effective teaching platform because it enables features that are impossible or hard to implement in the real world ([30], [29], [31]). For example, seeking to explain the effectiveness of teaching with virtual reality, [10] list four advantages: "experiencing," "engagement," "equitability," and "everywhere." First, in a virtual environment, the students experience the teaching, encouraging them to physically and mentally navigate the teaching content. [30] emphasize that new digital technologies can enable inquiry-based learning. There is convincing support that such active learning experiences are more effective than passive ones ([3]). The immersive aspect of virtual reality also increases the students' engagement—which is known to improve the learning experience and outcome ([12]). Furthermore, virtual reality has the advantage of equitability. It can allow access to the same material from everywhere with an Internet connection. In virtual reality, only people with eye disorders or who quickly experience cybersickness may be disadvantaged.

Of pedagogical importance is that virtual reality enables immersive learning experiences. From a learner's perspective, immersive learning can be defined as "the active construction and adaption of cognitive, affective, and psychomotor models through artificial experiences that are perceived as non-mediated" ([13], p. 3). According to [28], immersive learning comprises three design principles—realism, achievement, and presence—which play an important role in how effective learning can be. Realism refers to the "extent to which the environment in which you are immersed in is lifelike"; achievement refers to "the mechanism by which success toward performance goals is measured within the immersive learning environment"; and presence denotes "the extent to which the learner feels like he/she is connected or present immediately within the immersive learning environment" ([28], pp. 17–18). A key point is that if the learner feels immersed within the environment, the practice itself will be experienced more realistically and will be more tied to performance outside of the immersive learning environment ([28]).

Some studies have shown that virtual, immersive learning can benefit learning. For example, a recent field experiment by [35] showed that virtual reality reduced learning times by a factor of four compared with classroom training, and learners were four times more focused in virtual reality compared with eLearning. Also, some studies on virtual field trips found that virtual reality technologies increase learning outcomes and experiences ([25], [18]). Others have also documented disadvantages with virtual reality, such as extra administrational burden, technology constraints—particularly for real-time, interactive virtual reality teaching—and adverse side effects like cybersickness ([40], [45], [26]).

Despite the growing literature, there are several research gaps to be noted. First, academic evidence of the effectiveness of learning in virtual reality is still scarce ([31], [36], [5]). For example, [36] call for "future educational virtual reality applications should be more thoroughly evaluated by employing quantitative and qualitative research methods to assess the students' increase of knowledge and skills as well as the students' learning experiences (p. 22)." Second, there is also limited research that reports and evaluates real implementation in courses ([31]). Most studies only report technology development, explore opportunities, or report usability results of pilot projects ([36]). Third, the majority of studies focus on high-end virtual reality ([18]), although the simpler SVVR is more cost-efficient and easier to scale ([17]). Fourth—especially relevant to this study—there is a general lack of research on the use of SVVR in industrial engineering and management education (for an exception, see [26]). By studying the use and effectiveness of SVVR in operations management, this paper addresses these gaps.

3. App Development

To create a virtual factory environment, the authors teamed up with the Hilti Corporation, headquartered in Schaan, Liechtenstein, in the spring of 2020. Hilti serves the professional construction industry with high-end tools and services. Together, the team developed a virtual copy of one production area in Hilti's factory in Thüringen, Austria, shown in Figure 1. The app provides a general overview of the entire factory and a deep dive into one of its assembly U-cells as illustrated in the screenshot in Figure 2. In this cell, a handheld drilling tool for the professional construction market is assembled.

Graph: Figure 1. Hilti's Factory in Thüringen, Austria (Credit: Hilti)

Graph: Figure 2. Overview Scene Showing the Next One to Five Scenes in the FactoryVR App

To create the SVVR material, the authors used 360-degree photos and 360-degree videos captured with a low-cost 360-degree camera (Insta360 One). The content was carefully planned in workshops between the authors and Hilti. It was recorded during a one-day visit to the site. It covers a regular shift during assembly for a product in series production. Thirteen locations were recorded to capture the main flow of materials through the assembly processes. In addition, key information about the product and processes was collected and later blended into the virtual platform. The media content and information were merged on a virtual reality platform offered by immersive learning company Uptale. To ensure the FactoryVR app's quality and usability while also having a viable product ready for the fall semester of 2020, the app was developed using an agile working method over several sprints during a three-month period. Weekly Scrum meetings were organized between the teaching assistants and Hilti employees and course faculty.

To access the FactoryVR content, students use the technologies they already possess, such as smartphones, tablets, or computers. If teachers prefer 360-degree immersion, virtual reality cardboard or plastic headsets can be distributed. The app also works with high-end virtual reality technology (e.g., Oculus), but the relatively limited interaction possibilities of SVVR do not justify the investment and more complex administration in a course with many students. In the course reported here, all students received an inexpensive personal cardboard viewer, shown in Figure 3.

Graph: Figure 3. Teaching Staff with an SVVR Headset Using the FactoryVR App

With the cardboard viewer and a smartphone, students can "look around" in the factory by turning their heads and jumping from location to location. The 13 locations of the factory that were recorded are connected in the order of the production process via "previous scene" and "next scene" buttons. The learners "move" in the factory by jumping from one location to the previous or next location in the FactoryVR app. Alternatively, users can jump via a "back to overview" button to a map of the facility that shows all thirteen locations, which can be directly entered from this overview scene. On the smartphone, the different locations and the overview can be reached by focusing a few seconds on the respective button. On the computer, navigation is easily done with a mouse or touchpad. When users enter a location, a textbox appears that explains the operations that can be observed at this location.

The authors gamified the app in several ways. For example, gamified tasks include quizzes and short tasks such as calculating cycle times. The screenshot in Figure 4 gives an example of how a gamification task is overlaid on a scene in an assembly station. The gamified task in the example is to locate a specific part within 10 seconds (the purpose is to teach workplace organization). FactoryVR also contains blended information. For example, clicking the red camera symbol in Figure 4 opens a higher-resolution photo of the "one-point lesson" (a practice from lean production). Other blended information includes videos and overlaid text descriptions or explanations. For each action and correct quiz answer, the students earn a virtual "star," as shown in Figure 5. Playing the whole app and solving all tasks would earn the student 38 stars.

Graph: Figure 4. Example of a Gamified Learning Experience in the FactoryVR App

Graph: Figure 5. "Stars" Collected in the FactoryVR App (Here 22/38 Stars Are Collected)

4. App Implementation

The authors use the FactoryVR app in a graded student assignment in the three-ECTS credits course Production and Operations Management at ETH Zurich. This paper reports on the application and evaluation of the SVVR integration in the fall semester of 2020. That semester, because of COVID-19 restrictions, the faculty could only meet students virtually via online teaching platforms.

The course spanned 14 weeks and consisted of classroom lectures, independent study, and two separate assignments, each with a duration of three weeks. The first assignment commenced two weeks into the course and covered topics related to capacity management, planning and control, inventory management, and productivity. The second assignment was the FactoryVR assignment using SVVR. It began two weeks after the completion of the first assignment and concentrated on topics relating to lean production, layout, and process improvement. The assignments were conducted as group work outside of class. For both assignments, key concepts and theories were covered in class prior to and parallel to the assignments. The two assignments were self-contained and not overlapping in content. After the conclusion of each assignment, one class (90 minutes) was dedicated to presenting and discussing the answers to the assignments using a flipped classroom design. At the end of the semester, there was a final exam that assessed the entire course content, including the material from the assignments. The final grade for the course was a combination of the end-of-semester exam (weighted at 70%) and the two assignments (weighted at 30%).

The course was attended by 170 students—78% male and 22% as female—who performed the mandatory SVVR assignment in 43 groups: 41 groups with four students and two groups with three students. The groups were asked to respond to an assignment task that required them to visit the SVVR environment and collect relevant information. The assignment also encouraged the students to play all the gamified tasks, which would increase their general understanding of the factory and its operations. Table 1 provides an example of assignment questions and how they relate to the SVVR material.

Table 1. Three Examples of Assignment Questions

Questions	Relation to SVVR
Search the factory for the following lean methods: (a) Standard operating procedures, (b) Two-bin system, (c) Karakuri technologies, (n) ... Please note where you found them, include a screenshot, and explain their purpose.	The students had to find the listed methods in different parts of the SVVR factory. The students have to know what to look for, find it, and explain the purpose in the setting it is used.
Find three different types of waste ("muda") in the factory. Please note where you found them and include a screenshot.	The students have to search the factory actively for improvement potentials and wastes.
Create a Value Stream Map (VSM) of the current processes in the assembly cell. t.	The students had to use the information in the SVVR to draw the VSM (an additional table with cycle times was provided).

Each group was given a unique access key to the SVVR app, which enabled the authors to collect login and activity statistics. The student groups had to visit the SVVR app to answer the questions in the assignment. For this, students used both the handed-out cardboard glasses with their smartphones or computers. The computer interface was generally preferred for the collaborative part of the work because it enabled several members to navigate the same SVVR material—even when remotely cooperating via videoconferencing.

After the assignment, the teaching staff collected user statistics from the SVVR platform. Collectively, the groups started 1,050 new sessions and collected 8,878 stars. Each of the 43 groups spent, on average, 8 hours in the SVVR environment during the assignment. The variation in time spent in SVVR ranged from 2 to 19 hours, with a standard deviation of 3.0 hours. Overall, the user statistics suggest that the FactoryVR app was an important part of the course for the students.

5. Analysis and Results

This section reports the results of three different analyses the authors used to assess the relationship between the SVVR assignment and students' learning outcomes and experiences. First, the results from a regression analysis testing the effect of SVVR on learning outcomes are shown. Second, descriptive statistics from a postassignment survey that measures the students' learning experiences are presented. Third, key points from a qualitative analysis of course evaluation forms are summarized.

5.1. Effect of SVVR on Learning Outcomes

To test whether the SVVR assignment creates an effective learning environment, the authors assess if the groups that spent more time in the FactoryVR app—and engaged more with the virtual content—scored better on the assignment than groups that spent less time in FactoryVR. This choice is informed by the pedagogy literature that upholds "student time on task" as one of the most critical variables affecting learning outcomes ([41]). It suggests that the more time students spend engaging with a learning task, the more they learn. If the SVVR app was an ineffective learning tool, the students would assumedly not improve the learning outcome by spending more time on it. This association is tested with the following ordinary least squares regression (OLS) model on the group level:

LearningOutcomei=α0+β1Durationi+β2Engagementi+β3GroupStrengthi+εi, (1)

where LearningOutcome is the student group's assignment score in points; Duration is the amount of time each group spent in FactoryVR for active days; Engagement is a control variable for the level of engagement measured by the average number of scenes groups played on active days (i.e., days where a group spend at least 30 minutes in FactoryVR); GroupStrength is a control variable for the student group's performance on a previous assignment in the same course. The variables were standardized.

As explained, the course consisted of two graded group assignments, where the SVVR assignment was second. Both assignments were measured in grading points (0–100), which were translated into assignment grades. Duration is calculated as the sum of minutes that students in one group collectively spent in the SVVR app for every day they spent 30 minutes or more. Days, when groups spent less than 30 minutes in total in the app, are ignored to not capture times when the app was only tested and not used for dedicated assignment work. The results are shown in Table 2. Model 1 shows the results of regressing only duration with learning outcomes, and Model 2 includes the two control variables.

Table 2. OLS Results

Variable	Model 1	Model 2
Constant	0.000	0.000
	(0.151)	(0.133)
Duration	0.255*	0.342**
	(0.151)	(0.139)
Engagement		0.463***
		(0.138)
GroupStrength		0.150
		(0.136)
N (groups)	43	43
R2	0.042	0.307
Adjusted R2	0.065	0.254

1 *p < 0.1; **p < 0.05; ***p < 0.001.

As can be seen in columns 2 and 3 of Table 2, the groups' learning outcomes are significantly related to the time the groups spent in SVVR, indicating that the SVVR environment assists students' learning. The R2 numbers of column 3 indicate that the model's goodness of fit improves sharply when the controls are included. The model explains 30.7% of the variation in learning outcomes, which is reasonable and acceptable ([9]) given the context. In this model, the duration spent in SVVR remains statistically significant, and the level of engagement in SVVR is also significantly related to performance. The results for the group strength are insignificant, meaning that performance in the SVVR assignment cannot be explained by some groups being generally better in terms of academic performance than others.

5.2. Assessment of the Students' Learning Experience

The authors assessed the learning experience by implementing a postassignment survey. The students were asked to rate how they experienced the SVVR assignment on a range of learning experience variables. The survey was administered through an online survey platform. Table 3 includes the variable list, definitions, and correlations.

Table 3. Survey Variables, Questions, and Correlations

	Variable	Question	1	2	3	4	5	6	7	8	9	10	11
1	Meaningfulness	Relevant and representative of realitya	1
2	Usability	Ease of use after the first rounda	0.352**	1
3	Usefulness	How much you learneda	0.493**	0.335**	1
4	Enjoyment	How enjoyable it was to use ita	0.488**	0.377**	0.398**	1
5	Likelihood of change	Whether you will use what you learneda	0.371**	0.329**	0.519**	0.435**	1
6	Self-paced learning	I could go through the learning process at my own paceb	0,177	0.232*	0,175	0.221*	0,144	1
7	Increased interest	The virtual reality assignment has increased my interest in the topicb	0.385**	0.359**	0.462**	0.396**	0.458**	0,063	1
8	Explainability	After the virtual reality assignment, I am able to explain the learning content to a friendb	0.191*	0.268**	0.314**	0.317**	0.390**	0,176	0.405**	1
9	Memorability	I will remember the learning content longer than if I did not see it in virtual realityb	0.237*	0,177	0.210*	0.289**	0.383**	0,173	0.333**	0,098	1
10	Immersion	I could immerse virtually into the real factoryb	0.469**	0.339**	0.407**	0.453**	0.391**	0.267**	0.354**	0,133	0.442**	1
11	Learning experience	How would you rate your overall learning experience with the virtual reality assignment?b	0.434**	0.471**	0.435**	0.541**	0.465**	0,166	0.495**	0.319**	0.390**	0.548**	1

2 aScale: 1, poor; 2, fair; 3, satisfactory; 4, very good; 5, excellent. 3 bScale: 1, strongly disagree; 2, disagree; 3, neutral; 4, agree; 5, strongly agree. 4 **p > 0.01; *p > 0.05.

To avoid performance bias, the survey was launched and closed before the grades had been distributed. To avoid desirability bias, the survey was fully anonymous, which did not allow for tracking responses to individual students in any way. A consequence and limitation of this design is that the authors could not match the individual-level survey responses to the group-level app usage data or assignment grading scores. Before the deadline and after three reminders, 115 of the 170 students had filled out the survey. This corresponds to a 68% response rate, which is arguably a high number for fully anonymous student surveys.

One concern could be that nonrespondents were more negative (or positive) toward the SVVR assignment. To check for nonresponse bias, respondents were split into subgroups of early (n = 53) and late (n = 62) respondents, where the cutoff was set at the date of the first reminder. The authors checked if there were significant differences in the mean for key variables between the early and late respondents. The results showed no evidence of nonresponse bias.

The descriptive results are shown in Table 4. It presents support that the SVVR assignment was rated as providing a positive learning experience among the students. The mean is well above 3.00 ("satisfactory" or "neutral") for all included variables. The mode is always four or five. The average score is highest for "self-paced learning" (4.63) followed by "overall learning experience" (4.29). The overall learning experience is rated as five (excellent) by 41% of the students and four (very good) by 46%. Only 10% rated the SVVR experience as three (satisfactory), 1% as two (fair), and none of the 110 students who responded to this question rated it zero (poor).

Table 4. Descriptive Statistics of Students' Learning Experiences

	Variable	Minimum	Maximum	Mode	Mean	Standard deviation	Variance	N
1	Meaningfulness	2	5	4	4.05	0.76	0.57	115
2	Usability	1	5	4	3.69	0.88	0.77	115
3	Usefulness	2	5	4	3.92	0.80	0.65	115
4	Enjoyment	2	5	4	3.97	0.95	0.89	115
5	Likelihood of change	1	5	4	3.76	0.85	0.72	115
6	Self-paced learning	3	5	5	4.63	0.54	0.29	115
7	Increased interest	1	5	4	3.94	0.88	0.78	115
8	Explainability	2	5	4	4.25	0.63	0.40	115
9	Memorability	1	5	5	4.05	1.06	1.12	115
10	Immersion	2	5	4	3.82	0.86	0.75	115
11	Learning experience	2	5	4	4.29	0.68	0.46	110

The survey included four additional variables used as controls: gender, type of study program, experience with virtual reality, and working experience in manufacturing. Splitting these variables into meaningful binary categories and t testing for differences in the overall learning experience resulted in no statistically significant results. This suggests that the results are not primarily driven by a bias in the respondent pool's characteristics. Taken together, there is convincing evidence that the SVVR assignment provided a learning experience that was highly appreciated by most of the students.

5.3. SVVR Assignment Feedback

The quantitative analyses reported above show that (1) the groups that engaged more in the SVVR assignment scored better grades than those that engaged less, indicating a positive effect of SVVR on learning outcomes, and (2) the students reported an overall positive learning experience. To understand and explore the factors that drive these relations, the students' written feedback in anonymous feedback forms is analyzed. These qualitative statements stem from two sources: an open-ended question in the survey and an official course evaluation administered by the university.

The first source is the students' answers to the open-ended survey question, "Do you have any suggestions for improvements?," with 53 of the 115 survey respondents answering. Although the teaching team only asked for improvement opportunities, several students unsolicitedly praised the SVVR assignment. For example, students wrote, "It surprised me how good the immersion was even with the simple setup"; "It was a really fun experience being able to see how such a factory works with a bird's eye view"; "Overall, a really good experience that taught a lot!"; and "I tremendously enjoyed the walks through the workshop." Although these quotes are anecdotal, they provide additional evidence that FactoryVR was well received.

Most of the students' answers regarding improvement potential concerned technological limitations that reduced the level of immersion and engagement. The mentioned limitations include the following: motion sickness, no ability to move freely in the virtual app, poor visualization of navigation options, temporary inoperability of the mobile app with iOS, limited information about the factory processes, and no possibility to interact with employees or a tour guide in real-time. Others expressed that the cardboard viewers were impractical or substandard, making students prefer the simple two-dimensional web browser option. The teaching team took note of the suggestions for future improvements.

The second source of anonymous feedback is the official university-administered course evaluation form sent to all students after course completion. This survey is administered by ETH Zurich's Department for Learning and Education Technology (LET). The evaluation generally concerns the whole course, but one question concerns the assignments, of which SVVR was one of two. The students are asked to rate the effectiveness of the assignments ("The exercises helped me to understand and apply the content of the lecture") on a scale from "not true" (one) to "absolutely true" (five). Eighty-one students answered this question (n = 81). It scored a median of 5 and a mean of 4.5 (standard deviation, 0.70). Even in the hypothetical case that students would rate the first assignment better than the SVVR assignment, this high score provides additional support to the hypothesis that the SVVR assignment relates positively to learning outcomes.

Also, in the LET-administered course evaluation, several students unsolicited mentioned the SVVR assignment in answers to an open-ended question called "Elements to keep." Except for a few critical comments about the inoperability of the SVVR app with iOS, which was a temporary technical issue subsequently solved by Uptale, several students highlighted the FactoryVR assignment as an enjoyable learning experience, as illustrated by the following statement from one of the students: "Using the virtual reality glasses was incredible; bringing the students to a real factory environment to apply the theory in practice." Taken together, this second source of feedback is fully aligned with the survey the teaching team administered. It serves as another anecdotal evidence that the FactoryVR assignment also improved the learning experience.

6. Discussion

This paper has reported on a large-scale application of a tailored and gamified SVVR app for teaching production management. Teaching with SVVR is a new didactic approach allowing inquiry-based learning in a virtual environment. SVVR enables new teaching forms that have not been possible—or impractical—before. For example, the FactoryVR app goes beyond what a field trip offers and can provide access to multiple sites, far-flung places, and areas that might be unsafe for humans, such as a close-up view of a milling machine. SVVR helps the students relate to the course assignment through its immersive aspect and reduces the teaching materials' abstraction level ([12], [26], [31]).

6.1. Learning Effects of SVVR

Recent reviews of the literature have noted that it is difficult to assess the real learning effect of virtual reality due to limited research ([31], [32]; [36]) and due to mixed results ([45], [38]). Addressing this gap, the presented quantitative and qualitative analysis provides evidence that FactoryVR relates positively to both learning outcomes and learning experience. This research adds to the limited but growing studies in other fields that have assessed the effectiveness of SVVR. Noteworthy, in the study reported in this paper, all tests were carried out "live" in a course setting. Hence, it complements and adds to the studies that test SVVR in a laboratory or pilot setting, which constitutes the majority of studies in the literature ([31], [36]). FactoryVR has been continued at ETH Zurich. This study also reports effects on both the learning experience and learning outcomes related to SVVR. It also showed how SVVR could be gamified, which is known to assist the learning experience ([20]).

SVVR has some advantages compared with high-end virtual reality. Of particular importance is the fact that it can easily be scaled and used because students often possess the core technology hardware, which is smartphones or personal computers ([17]). Relatedly, costs can be kept low ([36]). A practical feature is the inclusion of students who would otherwise not participate in field trips such as factory visits due to other commitments, budget constraints, or sick leave. Provided that companies allow open access, SVVR apps similar to the one discussed here can be scaled across borders and eliminate travel time and budget constraints. It enables students to virtually visit factories located in distant countries without any negative impact on humans, the environment, or financial aspects ([26]). As demonstrated in FactoryVR, it also allows gamification of course content, which is an important topic in the learning sciences but has not received attention in the SVVR literature. Pedagogically, SVVR enables immersive learning ([28], [25], [18]) and inquiry-based learning ([11], [30], [17], [26]).

Although high-end virtual reality is superior in terms of immersion, it is much more difficult to manage and scale in large classes. SVVR provides a readily available alternative, which has received relatively less research compared with high-end technologies ([36]). The simple SVVR app presented here also differs from other studies where the virtual reality experience is bound to a classroom or laboratory setting ([25], [18]). Nowadays, anyone can make SVVR apps with a 360-degree camera and subscription-based SVVR app software, but, as [6] show, it will not be used unless teachers intend to adopt it.

6.2. Problems and Research Opportunities with SVVR

The findings also highlight problem areas that call for more research and experimentation with SVVR. Several of these problems are known in the literature and to SVVR users. First, one challenge is that, although SVVR is immersive, it is still far from the experience of a real field visit ([40]). In SVVR, students cannot touch, smell, or feel the real factory environments ([26]). It also does not allow the seamless walking around or real-time meeting functionalities of high-end virtual reality. More critically, recorded SVVR material does not allow students to engage or communicate with the personnel on the shop floor. Future research should seek to increase the feeling of immersion by lowering the economic and administrative bars related to adopting high-end virtual reality. A potentially groundbreaking development would be integrating real-time streaming technologies ([27]) with videoconferencing capabilities, allowing real-time virtual tours of real sites. Although this technology has found early use in real estate sales, it also has potential in education where real-time observations can add value—for example, in engineering and management education.

A second challenge is that some students experience cybersickness. This problem is known in the literature ([22], [26], [23]), but continuous technological development has reduced it. A related problem is the incompatibility of cardboard viewers with eyeglasses. In FactoryVR, both these problems are circumvented by encouraging the use of two-dimensional computer screens at the cost of immersion.

A third challenge has been technological hick-ups and errors that reduce the user experience. For example, inoperability of the software with certain versions of operating systems, too low computing power, or slow Internet access ([23]). Future research could also study the effect of blended information choices and gamification to suggest best practices for the SVVR design and content.

6.3. Limitations

This study on the use of SVVR in teaching has limitations. First, in fairness to the students, the authors decided not to run a controlled experiment where some students received SVVR, and others did not. Although this would significantly improve the research design, it would deprive some students of the SVVR experience. It would also be challenging to design a counterfactual that provided the students in the control group with a similar type of information and learning materials outside an SVVR environment. Second, although this study shows that SVVR can be a valuable and practical teaching aid, the data does not allow discerning why. Previous research has shown that factors like immersion and explorative learning are reasonable explanations, but more research is needed to conclude regarding SVVR. Third, this study's context relates to the physical world—factories—and it remains untested whether SVVR could effectively teach more abstract concepts and theories. These limitations leave promising opportunities for future research.

7. Conclusions

This paper presented the development, use, and evaluation of an SVVR app for teaching operations management. The app contains a virtual photo-realistic copy of a manufacturer's real factory, blended with information and gamified content. After implementing the app, the authors assessed the extent to which the SVVR app improved the student learning outcomes and learning experiences. It was found that students that engaged more in SVVR scored better on the assignment. It was also shown that students reported positive learning experiences. The SVVR app can be replicated and tailored to the needs of teachers in other fields.

Acknowledgments

We thank Gian-Andrea Gottini and Luca Solari Bozzi for assistance in developing the app and colleagues and students for valuable feedback on the VR application and assignment. We acknowledge the Hilti Corporation for the excellent cooperation and support—especially Kostas Kefalakis, Matteo Frigerio, Gernot Schubert, Melanie Kunde, and Dr. Christian Fornaroli.

By Torbjørn Netland; Rafael Lorenz; Daniel Kwasnitschka; Julian Senoner and Clemens Gróf

Reported by Author; Author; Author; Author; Author