What We've Learned About VR: A Brief Taxonomy of VR, Part 2

In Part 1 of this series, we discussed UI interactivity. In part 2, we will be taking a look at the use of placards and dioramas to convey information to the user.

Floating Placards, World Space

In Stay Safe with Horses VR, I experimented with placing large floating placards in midair, in a fixed place in the world, to help convey information.  It seemed to work quite well.

The nice thing about this is that they can appear and disappear without being an impediment to the space, and they can be placed strategically to tie them to specific concepts or locales. They can show up right when you need them, and then go away without cluttering up everything.

Of course, if they show up behind you, or if they show up edge-on, you don't see them, so they're a bit of a problem for "room-scale" VR setups. Similarly, if you walk behind them, they appear flipped, showing text backward, which is usually not ideal.

Another issue is that if they are placed in "world space", you can walk right up to them, which means either they get really blurry and grainy, or you have to present them at a very high resolution. Neither option is great.

PROS

• Simple and clear.

• Well-defined position, doesn't impede the player's sight when not needed.

•  Interact at a distance.

CONS

•  World space means it could be viewed awkwardly or not at all.

•  Requires high resolution since it can be viewed at arbitrary proximity, or else it is blurry and grainy.

SCOPE IMPACT: Low

Floating Placards, Button-controlled

The floating placards can be controlled by physical buttons in the 3D VR space.  This works like a remote control or a keyboard for a video screen, which is a very familiar interface convention for most people.

The placard itself, obviously, has all the pros and cons of just a static floating placard, but here, the need for a physical interface has to be considered; it's not just "point at it and click" on the placard, but a separate interface, possibly considerably removed from the placard itself, is needed.

Here's an example from the Pollinator Park app, where a button interface allows you to take a quiz. The questions are shown on the placard, and physical buttons are used to select an answer:

It works absolutely fine...when you're standing.  The problem comes when you are playing the game in a seated "stationary boundary" context. You're lower to the ground (because your head height is now your head height when sitting), and the UI controls are designed for a person standing.  There's also a fixed location you can "stand" to cause the quiz to activate. These two combined means a "stationary boundary" person (or perhaps a person in a wheelchair) has to lean side to side to see all three options for the quiz.

PROS

• Simple and clear. Intuitive for most audiences.

• Well-defined position.

• More tactile UI than buttons in speech balloons or placards.

CONS

•  World space means it could be viewed awkwardly or not at all.

•  Needs both 3D and 2D assets.

•  Requires high resolution since it can be viewed at arbitrary proximity, or else it is blurry and grainy.

•  Might need to adjust the scene layout significantly based on headset settings, user height, boundary context, etc., both to tie the 3D controls to the 2D presentation, and to avoid awkward visibility issues.

SCOPE IMPACT: Low/Moderate

Floating Placards, User-Space

So, how do you solve the problem of "They might be looking the wrong way" or "They may be in the wrong place" to see a placard?  One option is to attach the placard to the user's head so that it moves along with the user as they move and look around.  Here's an example from M-Care where a placard pops up summarizing a conversation you had with a dietician:

It certainly cannot be missed, and since it's attached to your head, you cannot not see it from the proper angle. And because it's always displayed at the same distance, the resolution doesn't have to be high.

The problem is that it's kind of jarring and off-putting to have something suddenly attached to your head, especially if it is right up in your face like this particular placard was. Because it's attached to your face, you can't lean over and try to look around it at something behind it, because it will always follow your head, and no matter where you look, it will stay right front and center.

And you can't solve that in your face issue without pushing the placard away from the player's head, which introduces another problem, namely, that the placard can start intersecting with geometry. Maybe there's a "sweet spot" where the placard is far enough away that it doesn't give you claustrophobic vibes and yet is still close enough that it doesn't run the risk of intersected geometry, but that's heavily dependent on the particular use case, such as whether your player can move or is ever close to a wall, another character, or other placards.

If you want to put buttons on it, that's another issue.  Things that are very close to your face are really difficult to click with your controllers.

PROS

•  Solves the "seeing it from a bad angle" problem.

•  Can't miss it.

•  Low overhead, reasonable resolution.

CONS

•  Jarring to have something attached to your head.

•  Can't miss it, but can't look around it.

•  Probably doesn't support interactive UI elements like buttons.

SCOPE IMPACT: Low

Floating Placards, Global Surround

So, if putting a video or image placard on a wall can be missed or be weird from the wrong angle, but putting one in user space can be off-putting and awkward, what's the alternative?

Well, one alternative is to absolutely surround the player with the video or image, so that it's everywhere, there's no bad angle, and nothing is attached to your face.  You replace the entire environment.

Here's an example from the Pollinator Park app:

The player presses a button, and you absolutely can't miss the video.  It surrounds you, and implies that the thing that's being talked about is important and majestic.

One down side to this approach, though, is that such a drastic change of environment immediately prompts you to look around, which disengages you from what's being said.  The first time I saw this, I was thinking, "Whoa, spaceship," and started looking around. I completely missed what was said and had to restart it to pick up the start of the spoken content.  If you decide to do this, it would probably be smart to lead with "throw away" text that's okay if the player misses it, or use a musical interlude to give the player time to reorient themselves in the new environment.

It also means the video has to be shot differently (or the 3D scene rendered differently) if it's going to be projected on a surrounding sphere.  This is much more complicated than producing a rectangular 2D image, but might be worth the effort.

It's also probably not a great solution for more mundane videos and animations; it's better for interesting, sweeping environment changes (like finding yourself in the microscopic world) than for more modest images (like showing a diagram of the parts of a horse).

Another issue is: where to place the sphere it is projected on?  Pollinator Park solves this by having you walk out onto a platform which gets surrounded by a sphere. To use this technique, you'd probably want a similar sort of setup; seeing the sphere intersect with the ground would be part of the distraction and make obvious how the transition was achieved, putting the focus on "I see how they did that" instead of on the content.

But it sure is effective when it happens!  There's probably a reason Pollinator Park saved it for a specific point in the "tour" rather than doing it for every little bit of video. They saved it for the space station video to give it a "wow" factor.

PROS

•  Biggest "bang" for presenting video and animations.

•  Can't miss it, but not in an intrusive way like user-mounted placards.

CONS

•  Needs very large resolution – it's large and in every direction.

•  Awe distraction, possibly detracting from content comprehension.

•  More difficult video / imagery production.

•  Imposes restrictions on the geography of the virtual space.

•  Could be a tonal mismatch for the more mundane images or videos.

SCOPE IMPACT: Moderate

Button-Driven 3D Dioramas

The Pollinator Park app has some interesting UI elements that prompt the user to tap different buttons to modify a diorama to convey concepts, especially about change over time.  For example, here is an example of a diorama of breakfast foods that change over time as you advance a simulation where pollinators go extinct and the effect on your food choices as that happens:

This idea of tying some buttons to a changing diorama seems like it could be used in a lot of situations, especially ones in which changes over time would be difficult to evoke in a real-life situation. Here's another clever example, where a display of different food products is shown, and you press a timeline button to show which ones disappear when the pollinators are extinct:

This is a clever way to tie temporal changes to physical changes. The approach works for several other contexts, too.  You could, for example, tap buttons to change between different species of animals or to slide in different maps.  Anything that has "diorama" like elements to switch between could benefit from an approach like this.

It does mean that you need to model the dioramas and design the button interface for them. The method of selection shown in the videos can be problematic in some contexts; for instance, the breakfast one is a lot easier when you are using "room scale" boundaries than "stationary" boundaries because you can more easily reach the buttons.

PROS

• Dioramas are effective in VR.

• Simple to understand for end users.

CONS

• UI needs to be kept relatively small and nearby.

• 3D dioramas are more expensive to produce than 2D floating placards.

• Some concepts probably wouldn't lend themselves to dioramas, so it may not be a general solution.

• Takes up space, so you need to plan a "location" for it in your VR space (or come up with a way to animate it in). This, in turn, might force a locomotion system where one would otherwise be unnecessary, which tend to be awkward or nausea-inducing.

SCOPE IMPACT: Moderate

Slider-Driven 3D Dioramas

Similar to the button-driven dioramas in Pollinator Park, Sea Level Rise Explorer: Long Beach has a slider-driven diorama.

Like the button-driven version, this is a fairly clear and accessible way to control state change in a set of things presented as physical objects.

This version allows more fine-grained control, as it is a continuous change instead of discrete values, but that also makes it feel more "fiddly".  It's difficult to pay attention to both where you're dragging the slider and the effect that's happening in the diorama.  It's fine for long-term trends across the slider, but it probably blurs small differences or important points in time.  For instance, this is great for communicating, "The longer this goes on, the higher the water level will be", but it would be a poor method for conveying important events that punctuate a timeline, such as on a political timeline.

PROS

• Dioramas are effective in VR.

• Simple to understand for end users.

CONS

• UI is fiddly; sliders are more challenging to control than buttons.

• Splits your attention.

• 3D dioramas are more expensive to produce than 2D floating placards.

• Some concepts probably wouldn't lend themselves to dioramas, so it may not be a general solution.

• Takes up space, so you need to plan a "location" for it in your VR space (or come up with a way to animate it in). This, in turn, might force a locomotion system where one would otherwise be unnecessary, which tend to be awkward or nausea-inducing.

SCOPE IMPACT: Moderate

Button or Slider-Driven Global Surround

The changes that a button-driven or slider-driven scene modify don't have to be dioramas; they can use the "global surround" idea mentioned earlier to allow you to modify the entire environment rather than just a diorama.  Here's an example from Sea Level Rise Explorer: Long Beach where you go from a "diorama" context to a "global surround" context for the modeled situation:

It's striking. Unlike with a video presentation, there's no problem with giving the player some time to adjust to the new surroundings; in the clip above, you see me turning to look around the blimp, but it doesn't get in the way of eventually settling back to explore the content.

Of course, changing elements all around you, in every direction, is almost certainly going to be more trouble than just in a diorama or a placard.

And these "global scale dioramas" also make it harder to know what to focus on as a player. Which direction should I be looking? In this case, it's probably a good idea to give the player some nudges to be looking in the right place.

PROS

• Being surrounded by the diorama might be more immersive than seeing it from above.

• Allows a global scale but letting player proceed at own pace doesn't cause "awe distraction".

CONS

• UI is fiddly; sliders are more challenging to control than buttons.

• Expensive to produce.

• Some concepts probably wouldn't lend themselves to global dioramas, so it may not be a general solution.

• Harder to prompt user focus.

SCOPE IMPACT: Moderate/High

See you in Part 3!

Written by John “CC” Chamberlin, Lead Developer