Center for Augmented and Virtual Reality

Edmund Lalor, PhD and Duje Tadin, PhD

A core principle of the Del Monte Institute for Neuroscience is that a fundamental understanding of the machinations of the brain is critical to develop and provide effective therapies for neurological and neuropsychiatric disease. In line with this, a major focus for the Institute—and a major strength—­is to elucidate the basic mechanisms of sensory, cognitive and motor functions in both neurotypical and clinical contexts. While this has long been the goal of neuroscientists around the world, a major shortcoming in tackling that goal has been the reliance on approaches using very simplistic and artificial experimental paradigms and stimuli that do not come close to capturing the complexity of the real world. To fully understand the brain, we must study how it functions under naturalistic conditions. The rapid recent maturation of augmented and virtual reality (AR/VR) technology promises unprecedented opportunities to accomplish this goal. Over the past few years, multi-billion dollar AR/VR investments by Google, Facebook, Apple, and other companies, have produced wearable, inexpensive and truly immersive technology that is certain to continue to improve and eventually become ubiquitous. The realism and immersiveness afforded by VR and related technology enables the development of much more realistic, multisensory environments that will allow basic scientists to study, at an unprecedented level of detail and sophistication, how information from different senses is integrated with motor and proprioceptive signals. This will enable us to understand how we perceive and engage with the 3D visual and acoustic structure of the environment, navigate through that environment, and make value-based decisions in realistic situations.

In addition to enabling significant advances in basic neuroscience research, VR and AR will facilitate the creation of virtual environments designed to study and treat a wide range of neurological and neuropsychiatric conditions. This could include research on disorders in which multisensory processing is specifically impaired, such as schizophrenia and ASD, as well as therapeutic applications such as the development of more effective cognitive training for MCI and AD, controlled introduction of social stimuli in ASD, and enhanced visual and motor rehabilitation in stroke patients.

Rochester's Position

The Del Monte Institute’s focus on AR and VR for neuroscience parallels a broader strategic push by the University of Rochester to leverage our expertise in optics, vision research, electrical engineering, and computer science toward significantly increased activity around AR and VR technology and applications. The University has recently implemented a pilot program for AR/VR research, funding ten pilot grants. Notably, over a half of the funded projects are on topics with direct relevance to neuroscience (e.g., cognitive training, multisensory processing, behavioral therapy, stroke rehabilitation, and 3D perception). There are current efforts to obtain external funding for an AR/VR training program and major research instrumentation for shared spatial audio and multisensory perception facilities. Most notably, there is strong interest to include AR/VR as one of the key areas of focus for the University’s Strategic Plan that is currently under development. This broader push to make UR one of the major players in AR/VR research presents an opportunity for a large number of synergetic connections with the Del Monte Institute for Neuroscience. Here, we would benefit from expertise elsewhere within the University, leading to technological innovations that are specifically designed to address neuroscience research and clinical questions.

Faculty Recruitment

It is critical to establish a leadership structure that will allow this initiative to move at a competitive pace. While we have a core set of researchers that already work in immersive environments and an even larger group of faculty that are considering this research direction, we do not have faculty who are primarily known for their work in AR/VR. Thus, first and foremost, we will work to hire an internationally recognized leader in the AR/VR field to drive the Center forward. For this hire, we will focus on areas where we already have a strong presence, namely multisensory processing and vision research. The rationale is that a strong research community of potential collaborators will make it easier to recruit a top researcher.