Revealing the Shopper Experience of Using a ‘Magic Mirror’ Augmented Reality Make-Up Application

Revealing the Shopper Experience of Using a ‘Magic
Mirror’ Augmented Reality Make-Up Application

ABSTRACT

INTRODUCTION

BACKGROUND

RESEARCH AIMS AND OBJECTIVES

THE MAGIC MIRROR MAKE-UP APP

SETTING

METHODOLOGY

FINDINGS

A user study was carried out alongside a larger evaluation
of the make-up app that the cosmetics brand, the AR
company and the store were conducting.

a. In-the-wild study

b. Survey

products. It seemed many people did not notice it at first or were not drawn by the brand logo and strapline “What’s your colour?” appearing on the tablets.
We also observed that those shoppers who did stop and look at the display did not subsequently interact with it.
As the stand-alone approach did not work, the shop assistants tried to entice passers-by to try out the app by telling them about what the virtual make-up app did and how easy it was to try.
The few times when visitors used the
app spontaneously without the encouragement of the shop assistant was when they saw other people who were using it, laughing or expressing admiration, interest or satisfaction.
As expected, the majority of people who interacted with the app were female.
On average, women used the application longer than the men did.
The level of interest from the shoppers who tried the application was very high especially once they realized what features the application offered.
It appeared that they found the app a convincing tool for trying make-up, seeing if it suited them and searching for the products they
liked.

a. Interactions with the Magic Mirror

b. Shopper experience

c. Survey results

d. Regression analysis

DISCUSSION

CONCLUSION

Virtual try-ons have recently emerged as a new form of
Augmented Reality application
Using motion caption
techniques, such apps show virtual elements like make-up or accessories superimposed over the real image of a person as if they were actually wearing them
However, there is as of yet little understanding about their value for providing a viable experience.
Our findings show that after the initial
surprise, the virtual try-on resulted in much exploration when shoppers looked at themselves on a display integrated in the make-up counter
Behavior tracking data from interactions with the mirror supported this
Moreover,
survey data measured perceptions of augmentation as well
as hedonic and utilitarian value of the app and suggested the augmented experience was perceived to be playful and credible while also acting as a strong driver for future behavior.
We discuss opportunities and challenges that such technology brings for shopping and other domains.

Augmented Reality (AR) has become increasingly available for end-consumers, mainly through smart device applications, but also through public interactive displays.
Contextualized information (e.g. arestaurant, a direction arrow, a figure in $$$) is typically overlaid on a view of the real world shown on a device display and captured by its internal camera as the user moves through a street or city.
A new kind of AR technology that is starting to be used as part of smart device applications is the “Magic Mirror”
The image of a person’s face, which appears on a device screen via the in-built camera (typically used for videoconferencing), is superimposed with add-ons such as make-up or accessories
In contrast to other AR apps that overlay the rear-facing camera image of the surroundings with digital information [3], the Magic Mirror uses the front-facing camera.
One kind of app using this technology is a virtual make-up try-on where the add-ons are created to realistically enhance the face; as far as possible giving the impression that one is truly trying on the make-up.
When the user moves their head, so too does the make-up by staying in the same place on the mirrored face. This illusion works through the application of motion capture techniques that build up an internal 2D model of a person’s facial features in real time.
However, it is not yet known whether this technique is effective in terms of ‘fooling’ users into thinking it is genuine and whether they would use it when selecting make-up in a retail store.
A question this raises is how convincing is the app to shoppers and does it entice them to try more or different kinds of make-up than they would otherwise?
Our research is concerned with investigating the uptake of this novel kind of AR technology in a real-world context. Specifically we ask: how does Magic Mirror as a new kind of AR application affect the shopping experience when in a public retail space?
A mixed method approach was used: in-situ observation and an extensive survey
Our findings showed that people did not simply walk up and use it but had to be talked through how to use it by an assistant or watch someone else before having a go.
We discuss how the success of these new kinds of AR technologies in a store depends on a range of factors, including whether the application is noticed, whether people feel comfortable trying it on, how long they use it for and what they do after using it
Finally, we examine the value of the Magic Mirror in a retail setting, in terms of its potential for enhancing the shopper experience versus the risk of it being perceived more as a playful gimmick.

a. wearables [6] have been dominating the research agenda. These include investigating the complete visual hardware pipeline from image capture and processing through to rendering and display. The technological advances of the last decade have seen all of these components coming together in integrated mobile devices

b. Ever smaller processors with greater processing capabilities, increased storage capacity, ubiquity of wireless Internet, mass adoption of smartphones and tablets and effectively unlimited storage capacity of on-line information have all contributed to the opening up of AR development and its commercial possibilities.

c. There are now software development kits (SDKs) available commercially that enable assembly of components within AR applications, such as AR recognition, tracking and content rendering (Vuforia, wikitude, D’fusion, ARToolKit or ARmedia)

d. Although these SDKs allow many companies to rapidly create novel AR apps, customized development of tracking and visualization components are often still required.

e. The areas where AR has seen most advances are tourism [18,30], aviation [10], culture [29] and education

f. A question for all of these domains is how does the AR technology change the user experience?

g. Does superimposing virtual information on a view of the real world on a display help people make decisions or enable them to understand better the context in real time? Is the way the information appears on the screen realistic enough and perceived as useful – in the way heads up display AR is commonly used in cockpits to help land planes?

h. A study by Kourouthanassis et al. [18] investigated the role of emotions in the adoption behavior of mobile AR systems for personalized tourist recommendations. They found that affect and arousal, as evoked by a system’s functional features, strongly impacted the user’s willingness to use it.

i. In the context of education, Chang et al. [7] have shown that an AR application that augments an art object with additional information can increase knowledge retention and deepens appreciation of paintings.

j. There is much interest in its potential for delivering an amended consumer experience, by which we refer to user experience that relates to consumption activities, both in public (such as retail) or private contexts (such as online shopping)

k. One of the first commercial applications was designed in 2008 for the car brand Mini, which presented a simulation of the car on a screen when a paper with corresponding trackers was placed in front of it [6].

l. Furniture brands, like Ikea, can now mimic pieces of furniture on a smart device screen as if it was literally placed in someone’s living room.

m. Huang and Liu [13] have shown that when AR simulates products such as furniture in a surrounding space, it creates a strong experiential value, especially when integrated in the consumer journey.

n. Companies such as Aurasma or LogoGrab have developed applications that augment products with 3D pop-ups and other visual content that appear when using AR tracking on a smartphone.

o. Other examples of AR in commercial settings include enriching surroundings with interactive displays or mirrors in a store. An example is an interactive wall display in a shop which shows snowflakes and gifts appearing as the shopper walks past it.

p. Another kind of AR app that has appeared for commercial purpose is the virtual try-on.

q. Early types of virtual try-on technology comprised either
a) avatar-based simulations where products are not tried on in real time on the users themselves but rather on a virtual proxy that resembles the user’s features and that the user can then manipulate [14,17] or
b) photo-based try-ons where products are tried on a user’s photo, which provides a static 2D experience

r. Studies of such virtual try-ons using virtual jewelry, make-up and clothes found that both hedonic and utilitarian aspects play an important role in the user experience [9,23].

s. However, in some cases the entertainment value can be a stronger cause for adoption of product virtualization technologies than usefulness, i.e. the more functional value

t. Personal characteristics of users, such as their openness towards novelty (typical for early adopters) [14] and body image [23], are important determinants of such try-ons’ perceived value

u. The users that are more curious about innovative technology (typically early adopters) would, for instance, pay more attention towards functional features and the application’s quality, while those with lower level of so-called cognitive innovativeness would be more likely to use it again if it was easy to use and playful

v. Here, we are interested in how people take to the Magic Mirror kind of AR, and more particularly, what they make of such an illusion

w. The goal of our research is to understand how it impacts the shopping experience, especially their initial perception, their willingness to experiment with products and the effect the experience of trying on different virtual make-up brands has on them.

x. we aim to investigate to which extent the levels of playfulness and convenience act as drivers for behavior when shoppers view the augmentation features to be credible.

y. Here, we investigate the use of the application in the wild in order to offer insights from a real-life context.

a. The aim of our research is to understand better the interplay between the new type of Magic Mirror AR technology in the retail context and reactions of shoppers towards it.

b. We were also interested in discovering whether
there were any unexpected effects or modalities of use

c. For this purpose, we investigated the types of responses that a tablet with the Magic Mirror AR elicited when situated at a make-up stand in a store and examined the forms of
interaction that emerge between shoppers, shop assistants
and this type of AR technology.

a. To attract the attention of shoppers, a screensaver displayed the make-up brand logo and the question ‘What’s your colour?” The rationale was that it would draw passers-by to the app, encourage them to start using it and try out the different colors of make-up.

b. Upon choosing a color or product, the virtual try-on mode is displayed, where the shopper can see his own face in the camera mirror with the addition of the selected make-up.

c. The application is intended for individual use and cannot simultaneously track more than one face.

a. The store has been set up to provide a futuristic style
of retailing where different areas present innovative
products or interactive technologies intended to offer new
kinds of consumer retail experience.

b. The Magic Mirror application was
installed on two tablets that were placed at eye level in the
make-up counter so as to be integrated with the process of
product browsing.

a. The researcher observed the visitors and made notes when
they interacted with the application on their own or with
others. Particular attention was paid to:

a) how visitors approached and interacted with the make-up app,

b) the most frequently used app features, c) visitors’ comments
and their bodily responses when trying out the app and d)
the follow-up behavior.

a. A 7-point Likert scaled was used for each statement, with 7
representing complete agreement with the statement and 1
complete disagreement

b. The sets of statements addressed:
a) consumer perception of AR features, i.e. augmentation;
b) playfulness;
c) convenience and
d) behavioral intentions

c. The initial objective of the survey was to
obtain opinions from Magic Mirror users over a longer
period of time than would have been possible to observe the
behavior in the store

d. Secondly, we also aimed to analyze to which extent the perception of such augmentation coincides
with the playfulness of the experience and, furthermore, if
that leads to behavioral intentions

i. Data from using the app features during the three months
period were analysed in terms of duration using the app per
visitor and different looks/products tried on.

ii. This data indicate that users spent considerable time looking at themselves with the virtual make-up and experimenting with different looks

iii. Next, we examine in more detail the way they approached
and used the Magic Mirror make-up app in terms of the
shopper experience.

i. Approximately 90% of the visitors were not sure what to
expect or what to do with the make-up app, so the shop
assistant told them to step closer, to touch the screen and
then select the product category or a color.

In general the visitors had no problems
using the interface.

Initial surprise:

When the augmented make-up first
appeared on a person’s face, the majority (around 80%)
showed surprise through their facial expressions, which
turned into delight when seeing, for example, a virtual
lipstick appearing on their lips or eye shadow on their
eyelids – exactly where it should be placed.

Convincing and realistic:

In most cases, it was found that the 2D tracking worked well: the virtual make-up appeared on the reflected face instantly, without delays, and
persistently followed the person’s movements
However, for some who had thin lips, the alignment was not quite right. Nevertheless, the extent to which this ‘off-tracking’ affected their interaction added to the experience rather than detracting from it.
The way the make-up stuck to their eyelids and lips and moved with them as they made these changes to their facial features was what was considered most striking.

Enjoyment:

The extent to which users enjoyed trying on the
make-up seemed to play a role in their continued use of the
app.
Occasionally, two people wanted to look at the screen
together. The tablet tracks only one face, however, and if
two faces appear on the screen at the same time the tracking
selects only one of them.

Men and children:

Around 75% of them felt compelled to state at the beginning that they
didn’t use make-up or made a gesture that this is not for them, but when the shop assistant remarked that many other men had already used it and that it was not real make-up,
they became more open to the experience and more curious.
they spent looking at themselves in the Magic Mirror app. They said
that they would never try real make-up on but that the
virtual one didn’t seem so intimidating and didn’t cause
them to experience feelings of social embarrassment.
The few children who tried it on also showed a high level of
curiosity and enjoyment.
While it was largely amusing for the men and children to
use the app, it also provided a new set of circumstances for
the women when they were shopping with their partners or
children

Follow-up behavior:

The presence of the shop assistant was important for follow-up behavior to occur.
She was able to point out to customers where a particular product they had tried on using the app was physically located on the make-up counter.

Based on this analysis, the survey items thus provided valid
measurement tools to how the respondents rated the
perceived augmentation, the playfulness, the convenience
dimension and conclusion of the shopping experience
Overall, the results showed that shoppers thought the app
realistically augmented their faces with virtual make-up in
real time
They also evaluated the experience to be very
playful and a large majority indicated intentions of future
engagement, such as subsequent use and talking to others
about the application.
The playfulness of the experience thus related both to the
enjoyment as well as to the creativity and exploration

We conducted a simple regression analysis to predict the following relations: the extent to which perceived augmentation predicts playfulness during application use and the convenience of it, as well as the correlations with behavioral intention.
From the first analysis (Table 2) we can observe that perceived augmentation acts a strong predictor of the playful experience that shoppers have with the application
Furthermore, both perceived augmentation and playfulness strongly correlate with visitors’ intention to return to the application for further use and to talk about it to others
When respondents perceive the Magic Mirror to augment their faces, they also perceive the shopping experience to be more convenient, as the app allows them to try on more products than usual and they feel less pressured to purchase them
than usual and they feel less pressured to purchase them. Both playfulness and perceived augmentation are relatively strong predictors of purchase intentions.
Furthermore, convenience strongly correlates with intentions to return back for future use.
These results demonstrate that as the level of perceived augmentation increases, so too does the user’s playfulness with the app and subsequently the likelihood that they will use it again, talk about it with others or purchase the tried- on products.
The increased levels of perceived augmentation are associated also with perceived convenience, which further implies future behavior, but to a lesser extent in comparison to the playfulness.

a. The observations, the tracked data and the survey data all indicate that the shopper experience with the Magic Mirror make-up app was engaging, often leading people to more experimentation with different colors for the make-up products.

b. It also helped some with decision-making when choosing or purchasing products.

c. However, because of the unexpectedness and novelty of the app, many passers-by did not notice it initially or appeared wary of trying it on in public. This suggests that simply placing a tablet with such an AR app in a store will not lead to people trying it by themselves.

d. Moreover, when placed in a store (rather than being an app a user downloads on his own device) that implies it requires someone in authority (i.e. a shop assistant) to legitimize a person using it in the store.

e. Also, seeing others using it can draw people closer and encourage them to take part. The role of the “honeypot effect” [4] is, therefore, even more critical for this kind of novel technology.

f. Once given the go-ahead, shoppers were happy to experiment and use it in the way intended.

g. Hence, far from being perceived as a gimmick, our observations showed that the people who tried the app perceived it to be convincing and useful. This was confirmed in the survey by the high scores for perceived augmentation.

h. Based on this data, it can be stated that the enhancement of the face through the Magic Mirror AR technology seems to create a strong perception amongst shoppers that the digital and physical elements are aligned and that the face is directly augmented with the virtual elements.

i. Our analysis also showed that shoppers experienced high levels of playfulness, excitement and surprise when interacting with the app

j. In some ways it is akin to McCarthy et al.’s [22] notion of enchantment, where the technology leads to a high level of absorption related to a state of concentration and attention.

k. For some of the shoppers, the app offered a different way of purchasing make-up.

Firstly, such an app included playfulness in the activity of make-up purchase.
Secondly, the virtual try-on allowed the potential customers to try on more products or colors because they could achieve this with a simple tap that takes considerably less time than trying on real products. The convenience allowed trying out colors that they would not otherwise have considered and thus permitted them to go beyond their usual set of choices.
Thirdly, such an app has the potential to change the way make-up is bought as colors can be placed on the face more realistically, while usual make-up testing consists mainly of putting testers on the hand and not on the face.

l. While technically more demanding, it would be interesting to see the effect of make-up being applied gradually, as if someone really is putting it on their face, mimicking not only the end result, but the process as well.

m. In terms of screen size, a tablet screen size actually offers an advantage, because fewer people appear in the camera view, thus making it less likely for the tracking to get confused and apply make-up to a person in a background.

n. Also, switching between AR mode and an app with products menu appeared problematic for some people. An alternative would be to keep the AR mode on all the time and allow the shopper to do everything (product selection, colour changing) in the same mode

o. This kind of Magic Mirror AR has much potential for other apps and settings, such as theatre, cinema, museums and art galleries, where dramatic, cinematic or historical looks could be experimented with.

p. In the context of health, enhancement with AR technology could show a predicted future image of the user, displaying potential changes that could occur due to healthy or unhealthy lifestyle choices

q. It could also be used in educational and training settings, providing make-up artists with a new tool to use when testing out their skills or perfecting new looks.

r. giving them a chance to try to create the look of celebrities. There is much scope for introducing a new level of realism and engagement into virtual try-ons

s. With new advances in 3D motion capture, it may also be possible to model the whole body, opening up opportunities for adding other features, such as tails, ears and hair.

a. The technology is capable of
creating an enchanting experience, whose multi-faceted
character comprises usefulness, realism, playfulness and an element of surprise.

b. However, in order for it to be
successfully deployed, seamless integration of the app as
part of the shopping journey is crucial

c. While this study shows the positive
reactions of shoppers to the app and builds on previous
research about experience of virtual try-ons, future research
can investigate how deployment and use of Magic Mirror
changes outside a store or when shoppers become famili