A Conversation With Newton

“Where did you conduct most of your experiments exactly?” I asked Sir Isaac Newton.
“I conducted most of the experiments here in my study, at Woolsthorpe Manor when Cambridge University had to close due to the plague,” Newton replied.
As someone who did not study the life and works of Isaac Newton prior to our fateful meeting, the answer was astounding to me. One of humanity’s most illustrious scientists made most of his world-changing discoveries while… remote studying? This was all the more telling because this first meeting occurred in the early winter of 2022, when COVID-19 strongly influenced university policy on in-person and remote learning.
“Did you enjoy studying here?”
“I think the study of hair can be fascinating, and studying hair care can be more cost-effective for some people…”

It was at that point that I decided to remove my Virtual Reality headset in amusement. I made note of the question I asked and the answer I received. It probably didn’t hear me correctly or heard “here” as “hair”. Or perhaps both. Don’t get me wrong, Sir Isaac Newton certainly did have an iconic hairdo and must have been eminently qualified to talk about hair care, but that was beside the point of the test.

The point was that we had just successfully used artificial intelligence to revive a historical figure, and I just had a chat with it in virtual reality.

VR? AI?

For those who may not be very familiar with some of these acronyms, here is the short version:

VR stands for Virtual Reality. It is a technology that fully immerses the user in a completely virtual universe. This is achieved by wearing a Virtual Reality headset, which fully obscures the vision and replaces it with a virtual environment that the user can interact with. (If you’d like to learn more about this, please check out our other articles on virtual reality.)

AI is short for Artificial Intelligence. According to John McCarthy, a renowned computer scientist and pioneer in AI from Stanford University, “It is the science and engineering of making intelligent machines, especially intelligent computer programs.” Stuart J. Russell and Peter Norvig, in their authoritative book on AI “Artificial Intelligence: A Modern Approach”, talk of “systems that think like humans. Systems that act like humans.”

This AI can take and has already taken on many forms in our daily lives:

• Robots that autonomously or semi-autonomously perform human tasks (how smart is your coffeemaker?)
• Cars that analyze their surroundings and use that data to drive themselves
• Video website algorithms that analyze what you’ve previously watched to recommend new content they think you would enjoy
• Shopping sites that show you what other customers “like you“ have also viewed and purchased

More recently, a variety of increasingly powerful AIs have taken center stage in the wider public sphere with their abilities to:

• Conduct seemingly natural conversations with humans
• Generate complex, technical writing
• Generate visual art, music, and poetry that try to be aesthetically pleasing to humans
• Write computer code
• …and many more things!

In comes Newton

The recent and very public ascendance of AI has created a lot of excitement and amazement, but it has also created a lot of chaos, noise and concerns. It has provoked a wide range of reactions, and the public is still grappling with many of its technological but also ethical (and at times even legal) implications, especially in education.

At KnowledgeOne, we firmly believe that instructors and students have to be at the center of any and all technologies. The technology must serve the instructor and student and not the other way around. For this reason, our first objective was to gain a better understanding of the “AI” tool: we must first understand how it is used and how it can be used.

In late 2021 and early 2022, the XR team at KnowledgeOne began actively investigating the capabilities of OpenAI’s GPT-3 AI model. We were particularly interested in exploring the potential use cases of GPT -3’s Natural Language Processing capabilities in combination with its massive amount of training data. The hypothesis was that its ability to enable unscripted, unrehearsed, and spontaneous conversations about specific topics could enhance the student’s engagement in a learning environment.

Given our expertise in developing Virtual Reality experiences for Concordia University, we immediately asked ourselves how these two technologies could complement each other. We amusingly called it the VRAI Project internally – if you are not familiar with French, vrai means “real,” – which is ironic given that neither VR nor AI are anything “real” in the physical sense of the word.

The answer to the hypothesis became the VR experience A Conversation with Newton, a fully immersive VR experience, where the user can visit Isaac Newton in his recreated study room at Woolsthorpe Manor and have an unscripted conversation with him, on any topic related to his life and work.

Main features of the experience

We proceeded with a three-pronged approach:

We leveraged GPT-3’s massive body of knowledge and asked it to embody a historical character: Isaac Newton

This means that the Virtual Newton will be able to answer, with a reasonably high level of accuracy, technical or factual questions about the historical Newton’s life, works and field of expertise. The range of topics can go from facts of life, such as date of birth, to more technical questions about equations and properties of the physical world.

A particular area of interest was asking questions about intent, where the Virtual Newton would provide insights on why the historical Newton may or may not have done certain things, all the while speaking in the historical Newton’s voice.

For example, when asked about the reason for developing calculus, the Virtual Newton consistently answered that calculus was created to better understand the motion of objects.

In order to ensure a historically authentic experience, the Virtual Newton was instructed to only answer questions related to Newton’s life and works, as well as his lifetime. Therefore, if asked about things such as the internet or Albert Einstein, the Virtual Newton will simply reply that it has never heard about these things or people.

This level of control is achieved through the prompt: the textual instruction that is provided to the AI in order to extract the desired behaviour.

We leveraged GPT-3’s Natural Language Processing capabilities to make conversation with this virtual Newton not only a possibility but a central feature

In order to make the interactions as natural as possible, we realize that the user must be able to ask any question they wish, with the wording they would naturally use. This means that the experience does not funnel the user into asking preformulated questions: the user is genuinely free to ask any question that they wish, and as long as the question falls within the limitations as defined by the prompt, the Virtual Newton will provide an answer.

The experience has built-in speech-to-text and text-to-speech capabilities to ensure that there is a two-way communication path between the user and the Virtual Newton.

We leveraged VR’s immersive ability to create a virtual “meeting space” between the user and the virtual Isaac Newton

The user is fully immersed in a 3D recreation of Newton’s study room in Virtual Reality. The team consulted photographs of the actual study room of Newton to recreate a historically authentic section of the manor. This not only immersed the user but also provided a sense of intimacy to encourage conversation.

Early designs

Final design

Our artists used historical paintings and busts of Isaac Newton in order to create a virtual avatar in Newton’s likeness.

Early designs

Takeaway

“If you had a chance to meet Isaac Newton, what would you ask him?”

Besides facts and information that can be checked with independent sources, some may ask: “How do you know that the historical Newton would have given this exact answer anyway?”

The honest answer to that pointed question is: “We don’t know. There is no way of knowing, and that’s ok.”

This specifically gets us to the heart of the subject. As much as the design was focused on recreating a historically authentic meeting with a famous figure of the past, the objective was never to recreate a virtual Newton who would be an exact clone of the historical Newton. We may get there one day, but we may never achieve that for the simple reason that the original (the historical Newton) is no longer with us to certify if the copy (AI Newton) is a faithful one.

We have noticed a common reaction of many who have tried this experience both in internal testing and public showcase events. They tense up, stay silent for a while and then declare with panic in their voice: “I don’t know what to ask!” Once the ice is broken, however, the conversations always flow.

A physics professor who tried the experience shared that she found its technical knowledge “very impressive”. A university administrator confided that she started feeling a sense of attachment toward the Virtual Newton after a 15-minute conversation with it.

We believe that there is a process of self-discovery that is happening with the users: they know the Newton before them is not real, and yet they are drawn into this conversation. The factual accuracy of Newton’s statements (largely very accurate) almost takes the backseat compared to the sense of wonder and connection the user develops towards the avatar. The virtual experience becomes a mirror, in a way, for the users to reflect on their curiosity and their views on artificial intelligence and virtual reality. The questions become almost more important than the answers themselves.

The avatar could have been Julius Caesar, as a fine arts professor suggested. It could have been Van Gogh. The experience would have surely been different but also similar in many ways.

The advances made on this project eventually allowed us to provide an AI case study to a course on Quantum Information Theory and Coding at Concordia University, where a small group of graduate students in computer engineering were invited to engage with an “AI Expert” on quantum computing (in the same way we did it to embody Newton, it was now an expert on quantum computing).

The students were then asked to reflect on their interactions and on how their field can influence AI or can be influenced by it. They were also asked to hypothesize on how this AI model was created, how they would construct it if given the chance and how it could be improved. It was an enriching learning opportunity that allowed the students to interact with an example of cutting-edge technology that is related to their field.

Future steps

From the “simple” perspective of combining VR with AI, the potential for education, when used diligently, remains impressive. From the personalization of the learning experiences to creating a level of interactivity way above the text or video formats, the VR AI approaches can develop emotional connections and turn learning experiences into lasting memories. On top of these, the instant feedback provided can further increase the likelihood of attaining the learning goals. AI can also analyze data collected from student interactions with the VR environment, providing educators with insights into student learning patterns and preferences, which can inform instructional decisions and program development.

All in all, AI combined with VR can provide a powerful tool for enhancing educational experiences, making learning more engaging.

However, despite the encouraging results of our use cases, we must remain restrained in our enthusiasm and excitement for this new technology, all while not ignoring the concerns that the educational community has about the democratization of AI. Our principles remain the same and focus on placing the instructors and students at the center of all our educational experiences.

The creation of A Conversation With Newton has undeniably allowed us to gain a better understanding of the potential use cases of artificial intelligence in an educational setting.

Disclaimer: Please note that KnowledgeOne is not in any way affiliated with OpenAI.

Author:
Ping Ng

Learning Experience Designer @KnowledgeOne.
Artmaker. Tinkerer.
3D Printing enthusiast.
Gamer. Night owl.