From Paper to Pixels: Transferring Handwritten Note-Taking Into Virtual Reality

While in the past, Virtual Reality and Augmented Reality needed hardware dedicated exclusively to one or the other, newer Virtual Reality headsets like the Meta Quest 3 combine both functionalities in one. This opens up possibilities to more easily implementable applications that use a combination of VR and AR, namely Augmented Virtuality, where a mostly virtual world is augmented with parts from the real one. We explore the possibilities that Augmented Virtuality can offer to enhance VR applications both from a theoretical perspective on the example of VR in driving automation and by offering a concrete prototype for note-taking in VR. This prototype, PaperVR, uses augmented virtuality concepts to enable handwriting on real paper inside a virtual environment. For that purpose, the front-facing cameras of a Meta Quest 3 were used, together with the MetaXR plugin, to create a tracked passthrough window, revealing the physical paper inside the virtual environment. As a point of comparison, a second prototype was developed which is intended to represent the current standard in handwritten VR note-taking, namely tablet writing. A user study was conducted, to compare both prototypes with each other and their equivalent writing methods outside of VR. It shows that paper writing is superior to tablet-based writing in VR for a synthetic task in our study. Additionally, in a practical note-taking task, it was possible to reach the same objective results inside VR as using physical writing outside a virtual environment, both regarding the number of correct answers and the answering speed per question.