There is no mixed reality!

What? But I thought…

Mixed Reality is kind of starting to replace AR one by one for some, for others it´s a third kind besides AR and VR. I thought I´ll add my point of view to the discussion today and you will understand why there is no mixed reality by itself.

In 2016 everybody is big into VR and claims to have known it for many years. Then someone writes down “we do VR” – adding “yeah, yeah, we also do AR and MR” having a nice triplet of buzz words in a row: VR/AR/MR. Microsoft labels HoloLens Mixed Reality and Magic Leap sometimes knocks it up a notch calling theirs cinematic reality. No definition is carved in stone or can claim the throne. In the end I also don´t care about names, but the result.

But, hey, why not let´s make a detour to history for starters. Paul Milgram published a paper in 1994 with his colleagues Takemura, Utsumi and Kishino called “Augmented Reality: A class of displays on the reality-virtuality continuum”. If you haven´t read it – go now, print it and put it up on your bedroom walls to really digest it, damnit! Here they already nicely describe visionary how the ultimate computer could look like and what kind of forms could exist blending reality levels. Their range goes from left (100% reality) to right (100% virtual) in their RV-continuum, know also known as the Milgram Mixed Reality Continuum. All stuff we market today can fit into this spectrum. It does not matter if you see the reality or virtual world through video-see-throughs, a window or on the screen. That´s left to other terms (they also describe). To sort things they asked:

Is the environment being observed principally real, with added computer generated enhancements? -or- Is the surrounding environment principally virtual, but augmented through the use of real (i.e. unmodelled) imaging data?

Obviously this leads to the categories we all talk about – AR and VR. Augmented Reality will let you remain in the real space with added information. It does not really matter (for this definition) if you just have a HUD system like Google Glassholes or look at the augmented content through a window via a tablet or phone. The visual quality may also vary from simple wireframes (like in the Sword of Damocles HMD from the ´60s) to high-end graphics we expect from Magic Leap. If you augment the user´s real world view with additional info, it´s AR. The term got a bit damaged by silly marketing apps with bad quality or boring overlays, but still, I´d say it´s the right term for what we expect from glasses like the Hololens or Magic Leap.

On the other extreme, Virtual Reality is well-known simple to define. You just see virtual, computer-generated (or digitalized) content without seeing anything of your real physical world you move inside as a human. Any game for the Vive, Rift, Gear, PS VR or Daydream is VR. Easy. But what if we see the chaperone system (to protect us from bumping into our furniture by displaying a grid or the outlines of the real world´s object) or if a real person joins into our VR world as a 3D-representation (scanned by a Kinect or similar)?

Well, then it´s time to talk about Augmented Virtuality. It fits into the continuum above inbetween the extremes. The naming get´s kind of tricky. It´s rather an augmented Virtual Reality. Though the “augmented” term would now refer to real world objects, while “augmented” in AR refers to virtual objects. But the concept gets clear: if you are in VR and add physically present objects it becomes something else than VR – you move left on the continuum line.

So, is this all there is?

viraugmixmodmediated_reality

This definition sure covers nicely the transition from real to virtual and it´s mixings. But right now we only talk about adding things to our viewport in AR or VR. While I will leave out commenting on other sensory input (augmented audio, etc.) there is more than the mixed reality continuum. Technology can add things to our view, subtract information from it or manipulate our perception of reality as a whole. Here the MR space gets limited by it´s default definition.

Systems to diminish reality could surely be seen as a sub-part of AR (to remove real world objects in your real world view). I´ve presented demo here and here long ago. But seeing the user´s perception of the world being altered, maybe not only through human sensory input, but also to the brain – how the world is perceived, another more abstract term seems useful: mediated reality. (See graphic, taken from wiki.) Once MR and AI connect us even deeper into the digital world – and all gets a bit more scary – I`m sure the term will have it´s rise (get ready for the discussion).

So, is there a mixed reality?

As said, it´s just names. Some people use the term MR to say that they create youtube videos showing a real person in a virtual reality environment by means of camera tracking and a greenscreen (this would be an Augmented Virtuality presentation of a Virtual Reality experience for me). Others use it to talk about AR “with convincing integration”. I´d stick with the phrase of Augmented Reality for that (not only because it better matches my domain)! It just makes more sense. The continuum describes all possibilities!

Microsoft did well in calling their overall approach Mixed Reality to begin with. The Hololens sure is a pure AR device. But with Windows Holographic and new devices around the corner that can switch between AR and VR – it was smart. They just put a new video online describing their range, too (a second video continues explanations):

So, is there a mixed reality? With the “old school” definition there is only a mixed reality continuum. You cannot be in a so-called MR experience. You can mix different realities – a virtual one, the real one, virtual pieces or real pieces to your current AR, VR or AV scenario. Some say “but MR has higher immersion than AR”. I disagree. Immersion comes through the story, not the visuals or the technology. A book can be way more immersive than Magic Leap. But, hey, let´s talk about immersion and presence another day. :-)

So, however we call it, Enhanced, Augmented, Modulated, Mediated, Mixed… Let´s work together on a great new computing era and enjoy the blend of flavours!