Augmented Reality (AR),Virtual Reality (VR) and Metaverse in Bioinformatics

Augmented Reality is a fundamental technology that will enable a significant paradigm shift in the way consumers interact with data, and it was only recently identified as a potential solution for a variety of critical demands. Enter augmented reality (AR) technology, which can see data from hundreds of sensors concurrently, superimposing useful and actionable information on your environment via a headset. With the rapid development of 5G, augmented reality technology becomes significantly faster and more data-rich. With its ease of use and accessibility for a range of diverse purposes (other than video gaming), widespread adoption appears possible.

Virtual Reality (VR) is a term that refers to a computer-generated 3D virtual environment into which users are immersed, as opposed to a 3D rendering of a 2D display.

Users can see the internal complexity of data in this context, obtain a deeper grasp of the relationships between various pieces, and discover previously overlooked connections between them. For these reasons, virtual reality has the potential to boost abstract information visualisation, and numerous studies have demonstrated that consumers comprehend data more quickly and easily in VR environments than they do in conventional 2D and 3D desktop display.

Modern VR is based on a low-cost stereoscopic head-mounted display (HMD), such as the Oculus Rift, Google Cardboard, or HTC Vive, that provides distinct images to each eye to create a three-dimensional sense of the scene. In the case of Oculus Rift, the head-mounted display is complemented by a head-tracking system comprised of an infrared camera, a three-axis gyroscope, and a hand-tracking device such as the Leap Motion. These advancements in virtual reality make biology visualisation extremely appealing because they allow consumers intuitive control over studying and altering complex biological data. Unsurprisingly, virtual reality has been extensively used to visualise biological data, including biomolecular and metabolic networks, microscopy images, protein-ligand complexes, biological electron transfer dynamics, whole genome synteny, and whole cell synteny, as well as medical education and research.

Bioinformatics research generates enormous amounts of diverse data. This treasure trove of data contains information about metabolic mechanisms and pathways, as well as proteomics, transcriptomics, and metabolomics. Often, in Bioinformatics, visualisation and study of associated structural – typically molecular – data is critical. For decades, technologies linked to augmented reality have been created and utilised to Bioinformatics challenges. Often, these technologies give “simple” visual help for the study, like when exploring and engaging with a protein on a 3D monitor and suitable interaction hardware. Additionally, these approaches were previously limited to expensive professional visualisation facilities. With the development of new affordable and frequently transportable technology, there is a strong case for adopting comparable tactics on a daily basis for research. Visual Analytics has been effectively utilised to evaluate complicated and heterogeneous datasets for several years. Immersive Analytics is a novel technique that blends these approaches with immersive and interactive technologies. There is significant interest in the broad application of Augmented Reality (AR) and Virtual Reality (VR) in the sciences of bioinformatics and cheminformatics for the purpose of visualising complicated biological and chemical structures. AR and VR technologies enable breathtaking and immersive experiences, opening up previously untapped prospects for research and education. With immersive 3D medical imaging, AR may assist with anything from diagnosis to patient communication, collaborative pre-operative planning, and surgical procedures itself, by superimposing a 3D MRI/CT image onto the patient. The advantages of 3D medical imaging extend far beyond their financial value; they represent the future of treatment.

More recently, virtual reality technology has been utilised to visualise and interact with individual molecules on much more portable systems based on cellphones. NanoOne (by Nanome, http://nanome.ai) and Molecular Zoo have been designed to teach students how to manage biomomolecules efficiently and affordably. In a similar vein, Peppy is a virtual reality tool designed to aid in the teaching of the fundamentals of the structure of relatively small polypeptides to undergraduate classrooms. David R. Glowacki et al. established a framework for interactive molecular dynamics in a multiuser virtual reality environment from a complementary perspective with educational and scientific uses. This system enables the simultaneous visualisation and modification of complicated molecular structures (fullerene, carbon nanotubes, etc.) by several users in the same virtual environment. Finally, the TU Delft-developed MolDRIVE system enables interactive molecular dynamic simulations with user-supplied artificial forces. This could be advantageous for achieving specific configurations of the molecules or molecular systems being researched. MolDRIVE is compatible with a range of VR systems, including workbenches and CUBEs, although it is not intended for use with less expensive devices.

Numerous desktop software provide virtual reality molecular visualisation when used in conjunction with high-end virtual reality hardware (Oculus Rift, HTC VIVE, and Microsoft Mixed Reality). They can be extremely beneficial to expert/professional users, as they facilitate visualisation, analysis, and interaction. Generally, installation and efficient usage of these programmes are more complicated than those made for smartphones, as they frequently require other dependencies and training. The majority of them are built on top of the Unity3D game engine (https://unity.com/). These applications are generally well-optimized and quite stable.

This group’s most popular desktop programmes include the following:
Molecular Rift is an open-source molecular viewer that enables interaction with the virtual reality environment by hand gestures rather than traditional VR controllers. It is built on Unity3D and the open-source UnityMol package for visualising molecules in Unity3D. UnityMol is a molecular viewer that was created to operate with the Unity3D game engine (https://unity.com/). It includes code for determining the secondary structure of proteins, as well as a variety of viewing formats and options. It is built on the VRTK framework and is compatible with the HTC Vive, Oculus Rift, and Windows Mixed Reality headsets. UCSF ChimeraX is likewise built on the Unity3D platform. It is well-suited for visualising and analysing biomolecular structures (including multi-person virtual reality sessions), particularly proteins and nucleic acids, using virtual reality headsets such as the HTC Vive, Vive Pro, Oculus Rift, Samsung Odyssey, and Windows Mixed Reality. However, one of Chimera X’s primary drawbacks in visualising SARS-CoV-2 structures is that big molecular structures (more than a few thousand atoms) display too slowly and produce stuttering in the headset. VMD is a traditional molecular viewer with a slew of intriguing features and support for virtual reality. It is compatible with a broad variety of devices, including flat displays, six-degree-of-freedom input gadgets, and haptics attachments. Numerous general-purpose virtual reality toolkits, including CAVElib, FreeVR, VRPN, and VR Juggler, are well-suited for use with VMD. Recently, a freely available software pipeline for converting molecular structures from VMD to 3D objects in a virtual reality environment was revealed. Recently, a GPU-accelerated ray tracing engine called TachyonL-OptiX was developed with the goal of obtaining stereoscopic panoramic and omnidirectional projections compatible with VMD. This enables visualisation of large molecules using head-mounted displays such as the Oculus Rift or even Google Cardboard.

Parallel to advancements in technology, algorithms, and visualisation software, it is possible to use the internet to construct virtual reality scenarios using protein and molecular structures. It is now possible to construct web-based virtual reality experiences that can be accessed directly using standard web browsers. VRmol is a molecular viewer that includes certain analysis tools and a virtual reality mode of display. It is quite well documented and compatible with HTC, Vive, Oculus Rift, Microsoft Mix Reality, and other virtual reality devices that support WebVR (Microsoft Edge and Firefox Nightly, for instance). This tool is capable of reading structures from various databases in order to aid in drug design. ProteinVR is a recently launched 3D-VR molecular viewer that runs natively in modern web browsers, as well as on a range of electronic devices and virtual reality headgear. It makes use of BabylonJS, a lightweight alternative to Unity3D designed specifically for this type of application. ProteinVR also supports collaborative work via the use of public URLs. The code is freely available at http://durrantlab.com/protein-vr/, and it can be tested at http://durrantlab.com/pvr/. Additionally, Autodesk Molecular Viewer is a new web-based molecular viewer that supports virtual reality and includes supplementary analysis and editing tools. It is compatible with a wide number of devices. It enables the visualisation of enormous molecular systems, such as whole viruses. This programme may produce input for a variety of virtual reality headsets just by navigating to a shared URL. Regrettably, the authors of this software decided to discontinue development in 2018. iview is another web-based solution for online VR molecular visualisation systems that also works offline. Though the software includes a virtual reality capability, we were unable to use it. Furthermore, the server on which it is hosted appears to be unstable, as we were unable to access it during several of our testing. Prof. Ricardo L. Mancera’s group recently devised a method for visualising molecular dynamics simulations on a stereoscopic 3D display. Utilizing the Unity game engine and written in C#, this implementation enables virtual reality viewing of molecular dynamics trajectories generated using various software packages (GROMACS, LAMMPS, and NAMD) in an HIVE facility. Additionally, the application can be run on a Windows PC desktop for local demonstration and testing. BioVR is a platform that enables the visualisation of DNA, RNA, and protein structures in virtual reality utilising Unity3D and C# and Oculus Rift headsets. Finally, it’s worth noting EPAM Life Sciences’ AR and VR applications (https://www.epam.com/our-work/videos/vr-ar-molecular-visualization-apps).

In summary, AR/VR is an exciting new wave that will be coupled with Bioinformatics, where massive data repositories will enable an AR/VR lens into scenarios in ways that deliver near-instant insight at a previously inconceivable degree of detail. As a result, this special session intends to disseminate cutting-edge research in bioinformatics and augmented reality to both academia and industry. It enables specialists to share the most recent technological advancements. The following topics are of interest: novel augmented reality/virtual reality devices for bioinformatics, augmented reality-based data analysis for bioinformatics, 3D models of biochemical structures for augmented reality/virtual reality, and VR-based visualisation for bioinformatics. Generating three-dimensional (bio) molecular models Face tracking and modelling for augmented reality applications in the healthcare sector Interactive Molecular Graphics for Augmented Reality, augmented reality-based human-computer interaction in bioinformatics, three-dimensional genomic analysis Observing and interacting with protein molecules; Improving docking performance. The majority of bioinformatics workflows need the usage of a variety of tools. Often, this diversity necessitates data transformation, conversion to other formats, and transfer, resulting in a significant time and expense overhead. This overhead is especially noticeable in virtual environments, as the data to be immersively shown may occasionally require optimization by visualisation professionals to ensure compatibility with the virtual environment.

The metaverse will be the future.The metaverse is a type of mixed reality that is rapidly gaining popularity in consumer electronics. Not only will the combination of augmented and virtual reality add digital aspects into the physical world, but it will also merge the physical and virtual worlds. The Metaverse is a collectively shared virtual area that encompasses all virtual worlds including the Internet. While it may include replicas or adaptations of the real world, it is distinct from augmented reality. The phrase “metaverse” is derived from the prefix “meta” (meaning beyond) and the stem “verse” (a backformation of “universe”); it is often used to refer to a future iteration of the internet that is composed of persistent, shared, 3D virtual places connected into a perceived virtual universe.