Over the past few years, we’ve been exploring sound as a sensory channel for understanding the physics of real-time interactive molecular dynamics simulations. In the molecular sciences, sound is a vastly underutilized means for data processing, partly because audio representational standards are less well defined compared to graphics. For example, depicting an atom using a sphere is an arbitrary decision, but it’s intelligible owing to the fact that an atom and a sphere are both objects which are spatially delimited. Defining clearly delimited objects in the audio realm is not as straightforward, neither spatially nor compositionally. For example, it is difficult to imagine what constitutes an ‘atomistic’ object in a piece of audio design or music. Our work suggests that sound is best utilized for representing properties which are non-local: potential energy, electrostatic energy, local temperature, strain energy, etc. The non-locality of these properties makes them extremely difficult to visualize using conventional graphical rendering strategies (and even if there were effective strategies, would lead to significant visual congestion).

The video shows a real-time interactive simulation of 17-ALA peptide in which our resident sonification experts (PhD student Alex Jones, Prof. Tom Mitchell, & Prof. Joseph Hyde) track potential energy in real-time, in order to interactively generate sound. The video shows Alex manipulating a small protein from within Narupa, and highlights how the sound responds to dynamical changes in the potential energy. The sound changes as Alex takes the protein from its native folded state to a high-energy knotted state. Eventually, Alex unties the knot, and you can hear as the protein relaxes to a lower energy state.

New paper posted to the arXiv, which I’m really excited about, where we describe how user-guided real-time interactive quantum mechanics (QM) simulations in virtual reality (iMD-VR) can be used to train neural networks to learn QM energy functions. The paper is entitled “Training neural nets to learn reactive potential energy surfaces using interactive quantum chemistry in virtual reality”.

As far as I know, this is the first ever demonstration of real-time interactive QM in VR, something we were able to accomplish through collaboration with Markus Reiher and Alain Vaucher, our colleagues at ETH in Zurich, who worked with us to develop an interface between our VR framework Narupa, and their excellent SCINE quantum chemistry package. Silvia Amabilino and Lars Bratholm have have also done an excellent job making their excellent GPU-accelerated neural network (NN) framework for learning QM potential energy surfaces available as an open-source package on GitHub – see the paper for more details!

This is some fun stuff that we’ve been building in the Intangible Realities Laboratory! This video shows my perspective as I tie a knot in 17-Alanine using the Narupa framework while wearing the new customized Etextile VR glove prototype designed by Becca Rose & Rachel Freire. Alex Binnie did a great job building the software interface for the gloves. The gloves have been designed specifically to enable efficient pinching and grasping of the sort required in a VR-enabled real-time simulation environment. Stay tuned – we’ll soon be open-sourcing the designs, for both the gloves and for the software interface. Meanwhile, we’ve just posted to the arXiv a paper we’ve written about the design. Stay tuned for more!

Sounding the gong & ringing the bells the moment we set the project free!

After nearly three years of research & development, I’m happy to announce that we have finally open sourced our multi-person, virtual-reality enabled, real-time simulation framework under GPL. It lets groups of researchers go into VR, and touch molecular objects as if they were tangible objects. Our working name for the project is ‘Narupa’, which we arrived by combining the prefix ‘nano’ and the suffix ‘arupa’. Wikipedia explains how arūpa is a Sanskrit word describing non-physical and non-material objects. It seemed to us a good concept for describing what it’s like to interact with simulated nanoscale objects. It’s still a name-in-progress, so let us know how you find it.

We first prototyped this technology in Jan 2016 at London’s Barbican Arts Centre, as part of an ‘Open Lab’ residency which I organised using funding from the EPSRC and the Royal Society. It involved several participants from my University of Bristol research lab and also Phil Tew from my company Interactive Scientific Ltd., all of whom feature in a blog post & video which we made at the time. We’ve been hard at work in the intervening years, for example using it to carry out the studies described in our 2018 open-access paper in Science Advances. A number of my academic and industrial research colleagues have already joined us in our community efforts. I look forward to announcing a whole host of interesting partnerships over the next few months. It’s been particularly exciting for me to observe how our consciously open-source ethos has inspired my international research colleagues to release open-source versions of their own simulation codes, in order to participate in a community. We’ve had several emails from excited collaborators who have cloned the repo and are getting things running.

I’m absolutely delighted to have set this project free into the intellectual & cultural commons. Getting it out there has taken some doing, but now it’s done, set free into the commons as an open resource for enabling communities to cooperatively learn from one another. The video I’ve embedded in this post marks the moment at which we set Narupa free, complete with an ad hoc ceremony involving gongs and bells. It’s a credit to the fantastically creative VR researchers that I have the privilege to work with – Mike O’Connor, Alex Jones, Helen Deeks, Lisa May Thomas, Rebecca Walters, Simon Bennie, and Alex Binnie – these are people who know when to sound the gongs and ring the bells!

Stay tuned over the next few months, because there’s plenty more on the horizon. We’re building loads of cool new open-source features like quantum mechanical force engines, real-time data sonification and audio, and also code enabling you to stream your own real-time simulations from the cloud! We’ve also got a number of papers in the pipeline which will be coming out soon, where we will demonstrate a whole host of interesting application domains. We’ve even included instructions on how to build your own multi-person VR lab.

I’ve recently started a new community-focused project called SimuLitix. In what follows, I outline why I stepped away from my old company, how that experience informed the design and values of SimuLitix, and how I think the new company can productively work with the old company. My intent in writing this blog is to empower others to learn from my experience. It’s been quite a learning curve for me, especially useful in the way that it has evolved my own thinking regarding the intellectual commons – and I think others might stand to benefit from the thoughts and observations outlined herein.
Continue reading →

Really excited to report that the open access Science Advances paper published by O’Connor et al. during the summer, entitled “Sampling molecular conformations and dynamics in a multiuser virtual reality framework” has since generated significant media exposure, having been picked up by a number of scientific media outlets. Nature, the New York Times, and the BBC’s “Science in Action” show (the VR piece begins 7 mins in) all contacted me in order to discuss the implications this work could have for nanotech research. It’s been exciting to witness the interest which the paper has generated. It certainly seems to be captivating people’s imaginations, and is attracting lots of attention by workers across academia & industry.