immersive memory experience in mixed reality
Totem is an immersive experience that allows individuals to visualize memories in mixed reality, anchor them to objects in their home, and experience them with others.
Challenge Sponsor
Team
Mai Bouchet
Zihan Cheng
Emi MacLeod
Raj Makker
Shelley Xia
Project Timeline
9 weeks
My Role
Gestural Interaction Design
Interaction Flow
Final Presentation
opportunity
The Microsoft Design Expo is an annual design challenge for top design schools from around the world. This year, they posed the prompt "Intentional Design for Positive Cultural Impact in Mixed Reality" to seek out new application concepts for the HoloLens outside of gaming and entertainment.
Our team identified the home as an opportunity due to technology's challenge in breaking intimacy and human-to-human interaction, in particularly the private setting. Therefore, we reframed the initial challenge to:
"How might we overlay mixed reality in the home to promote human-to-human interactions and imbue meaning into the objects in the home?"
To better understand how best to intervene, we asked the question "What makes a home?". At the core, we came to define the home as the combination of people (relationships), collections (objects in our home), and memories (experiences). Totem lives in the intersection of these 3 crucial aspects of a home.
solution - design concept
Totem leverages the memories that people already associate with their objects in their home and allows them to share them with others in mixed reality.
Anchor memories to objects in the real world
Re-experience memories
Share or collaborate memories with others
STORYBOARDING as an ideation tool
Our team experienced a few creativity blocks during the ideation process and decided to each draw our own storyboards in order to flush our what we have in our minds. Through sharing our storyboards, we were able to arrive at a common theme and developed the following final storyboard:
After experiencing the shared memory together, the object is imbued with both the past but also their dsarecently-shared experience of going into mixed reality to revisit that memory.
dsadsa illustration by Shelley Xia
3d ui + gestural design
Experimenting with AR/MR UI
object-anchored ui
Our team recognised the benefits of familiarity and low learning curves with 2D icons and tried to find a balance between the two options.
We chose a ring UI around the cup with a familiar swipe gesture motion to balance an unique 3D UI with an easily-recognisable gesture.
dsa
hand gesture + selection feedback
For more precise interactions, we explored different methods of two-hand gestures to scale, move, set objects and environments. For quick exits and adjustments, we used one-hand gestures that are more intuitive and faster.
For user feedback, we imagined different methods of how an object lights up when it has been selected.
interaction flow
gESTURAL INTERACTION
Great for fast, scalable, and directional-driven input, we implemented this for features such as initiating the application, scaling totem from a globe to a fully-immersive experience, placing the memory into an object.
voice interaction
Voice and language input are imprecise but offer an infinite range of exploration and interaction, including the benefit of keeping the users' eyes focused on the experience and not an UI. For that reason, we implemented voice when the user is adding and editing to a memory.
Audio Feedback
Audio is one of the most important design aspects to keep a user feel immersed and present in an immersive environment. Therefore, audio output and feedback increases from external cues to fully immersive 360 audio as the user increases their immersion.
the experience
the what
Totem is a memory experience that is always initiated by the user. She can create a new memory, re-visit an existing memory, or share the memory and experience with others - all of which are anchored in an object.
When the user selects an object, a highlight will first appear around the object and the a ring UI would appear.
the how
To create a memory, the user can pull out the "add memory" icon. To experience an existing memory or to share them, she can pull out the other orbs. To add to the memory, the user can choose to add through her own digital archive (photos and videos) and/or pulling content from online.
There are 3 level of the Totem experience:
- A snow globe-like third-person view
- Mid-size globe in which the user can peer into
- Full 360 immersion
With each of these levels, we imagine that the audio will also match the scale - where the audio will become more immersive as they increase the scale.
the why
Totem makes it possible for us to experience and share our memories in entirely new ways - making memories both tangible and possible to share in a physical space. It also creates a new psychological dimension for people to think and interact with their possessions differently - one in which we hope will make the home more meaningful.
concept video
takeaways
- Multimodal interactions requires deep thinking on when to use what: Our team had many conversations about when the user should use gesture, an icon tap, or voice to understand how to leverage each interaction. Due to the lack of time to develop a comprehensive interaction model that encompasses all of these, we only dived to the level of broad generalisations. I am very fascinated about this - pairing which interaction for what intent - and am working on this for our NASA 2020 Europa Clipper Mission project.
- Not everything needs to be in 3D: For quick commands, 2D icons seemed to be more intuitive for our user tests. However, if the task is oriented around a 3D object, our users were delighted that their UI was now in 3D and leveraged the world at scale. Balance between the two was key.
- Ideation for design concepts require the "black hat" much later on in the process: This project was the ideal "think big" design concept project. Because of this, we were pushed to not consider implementation so to keep the "crazy ideas flowing". To make this happen, I learned that we had to intentionally push back our technical or business concerns until our concept was formed. A very different approach to design!