User experience evaluation

User experience evaluation (UXE) or user experience assessment (UXA) refers to a collection of methods, skills and tools utilized to uncover how a person perceives a system (product, service, non-commercial item, or a combination of them) before, during and after interacting with it. It is non-trivial to assess user experience since user experience is subjective, context-dependent and dynamic over time. For a UXA study to be successful, the researcher has to select the right dimensions, constructs, and methods and target the research for the specific area of interest such as game, transportation, mobile, etc.

Dimensions

There are many different dimensions to consider when choosing the best assessment approach:

* Goal: Summative (on the final product) or formative (during the process)
* Approach: Objective or subjective
* Data: Quantitative or qualitative
* Granularity: Momentary, episodic, or overall UX
* Setup: Lab or field

Laboratory experiments may work well for studying a specific aspect of user experience, but holistic user experience is optimally studied over a longer period of time with real users in a natural environment.

Constructs

In all cases, however, there are certain aspects of user experience that researchers are interested in (measures), and certain procedures and techniques used for collecting the data (methods). There are many measures and some high-level constructs of user experience that can be used as the basis for defining the user experience measures, for example:

# Utility: Does the user perceive the functions in the system as useful and fit for the purpose?
# Usability: Does the user feel that it is easy and efficient to get things done with the system?
# Aesthetics: Does the user see the system as visually attractive? Does it feel pleasurable in hand?
# Identification: Can I identify myself with the product? Do I look good when using it?
# Stimulation: Does the system give me inspiration? Or wow experiences?
# Value: Is the system important to me? What is its value for me?

To properly evaluate user experience, metrics and other factors surrounding a study need to be taken into account, for example:

* Data (metrics): The time taken to complete a task.
* Scale (metrics): Indicators that show effectiveness, efficiency and satisfaction.
* Other Factors: Conditions of use, the surrounding environment and other human factors.

Methods

An individual method can collect data about a set of specific constructs of user experience. For instance, usability testing is used to collect data about usability construct. Methods also differ if they are to measure a momentary or episodic experience (i.e., assessing how a person feels about a specific interaction episode or after executing a task) or an experience over time, also known as an longitudinal experience. UXA methods can be classified in three categories: implicit, explicit and creative methods.

Implicit methods

Implicit methods of UX research focus not just only on what the users say, but also on what the user cannot express verbally. Many available tools can assist in the implicit evaluation, in particular to gather implicit or objective data. When available, UX researchers utilize state of the art equipment to uncover all aspects of the experience.

Examples of implicit evaluation methods and tools:
* Eye tracking
* Attention tracking
* User tracking
* Task and reaction measurement, galvanic skin response or skin conductance
* Electroencephalography (EEG)
* Observation studies: participant observation where observers monitor the participant's reactions such as facial and other gestures, the tone of voice or other body language cues

Explicit methods

Explicit methods of UX research explore what the user is consciously aware of getting them to reflect on their own feelings or thoughts, and gather their views and opinions. An important aspect of explicit methods includes usability testing and emotion evaluation.

Emotion assessment

When investigating momentary user experiences, we can evaluate the level of positive affect, negative affect, joy, surprise, frustration, etc. The measures for emotions are bound to the methods used for emotion assessment, but typical emotion measures are e.g. valence and arousal. Objective emotion data can be collected by psychophysiological measurements or by observing expressed emotions. Subjective emotional data can be collected by using self-report methods, which can be verbal or non-verbal.

Examples of emotion assessment methods:

* Psychophysiological emotion measurements aim to identify emotions from physiological changes in muscles (e.g. face), pupils, skin, heart, brains, etc.
* Expression
* Think aloud protocol can be used for reporting emotions (real-time verbal self-report)
* Positive and Negative Affect Schedule (PANAS) (retrospective verbal self-report)
* Geneva emotion wheel (retrospective verbal self-report)
* Photographic Affect Meter (PAM)
* Emotion slider (continuous non-verbal self-report)
* Sensual evaluation instrument (SEI) (snapshot non-verbal self-report)
* PrEmo, a new version of EmoCards for assessing emotion (snapshot non-verbal self-report)

Creative methods

Equally important to implicit and explicit methods are the creative methods that the user researcher can utilize in order to bring together the design team's view, as well as the target market's dreams, aspirations and ideas of optimal design. These activities are more open and allow people to either co-create with the engineers/designers, or to use their imagination to express their ideal system.

Examples of creative assessment methods
* Co-design activities
* Creativity workshops
* Paper prototyping, wireframing, mind mapping
* Card sorting
* Personas

Longitudinal

In contrast to identifying a momentary emotion, longitudinal UXA investigates how a person feels about a system as a whole, after using it for a while.

Examples of longitudinal UXA methods (excluding traditional usability methods):
* Diary methods for self-reporting experiences during field studies
* Experience sampling method (ESM) for self-reporting during field studies
* Day reconstruction method (DRM) – story-telling to reveal the meaningful experiences during field studies
* AttrakDiff questionnaire for overall UX evaluation
* User experience questionnaire (UEQ) (available in several language versions)
* Ladder interviews – e.g. to find out attitudes or values behind behaviour or experience
* Holistic user experience (HUX) identifying the relevant product factors for holistic user experience

Areas of UXA research

Transportation

Automobiles have come a long way since their beginning in the late 19th century. One of the major things that have helped automobiles to provide more safety and convenience is electronics. With the advances in technology and electronics, car manufacturers have been able to offer a wide variety of services and conveniences. From the creation of the electronic fuel injection to the popular global positioning system found standard in many cars today, the auto industry has revolutionized the way people travel from place to place. Understanding how people interact with vehicles today, what contributes to a great driving experience, what is their current relationship with the car, what placement does it have in their lives, is key to the development of these technologies. This information ensures user-centered design practices to generate cohesive, predictive and desirable designs.

Once specific design concepts and ideas are on the table, UXA researchers further explore how people react to them regarding desirability, findability, usefulness, credibility, accessibility, usability and human factors metrics. Outcomes of this work includes user requirements, concept validation, and design guidelines. Researchers have conducted intriguing research to answer questions such as: could an In-Vehicle Infotainment (IVI) system with a speech evoked personality change your relationship with your car?, could an in-car system support unwinding after work?, could in-car solutions address the special needs of children as passengers, and assist the parents with the task of driving? and many others. Additionally, workshops and gatherings of researchers around the world take place to discuss current evaluation techniques and advance the field of experience research in the area of transportation. An important professional venue for this work is AutomotiveUI, the International Conference on Automotive User Interfaces and Interactive Vehicular Applications.

UXA methods for transportation

As with other UXA's the method chosen has a lot to do with the outcome desired and where the project is in its design cycle. Given that, methods are selected best suited to the research problem which most times ends up being a combination of implicit, explicit and creative. Some methods include:

* Interviews: both structured and un-structured.
* Diary studies
* Workload assessment questionnaires (i.e. DALI –Driving Activity Load Index adapted from NASA-TLX)
* Subjective assessment of interfaces questionnaires (i.e. SASSI—Subjective Assessment of Speech System Interfaces ) that can lead to design guidelines to speech interfacesAreti Goulati and Dalila Szostak. 2011. User experience in speech recognition of navigation devices: an assessment. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '11). ACM, New York, NY, USA, 517-520. DOI=10.1145/2037373.2037451
* Experience Probing (Prototypes, storytelling, storyboards)
* Co-design activities
* Observations (i.e. coding for frustration, delight and other non-verbal cues)

Video games

A relatively new pursuit in video game play-testing is UX and usability research. An increasing number of companies including some of the world's biggest publishers have begun outsourcing UX evaluation or opening their own in-house labs. Researchers use a variety of HCI and psychological techniques to examine the effectiveness of the user experience of the games during the design process.

There are also some companies starting to use biometrics to measure the relationship between in-game events and the player's emotions and feelings (the UX), such as Player Research and Serco ExperienceLab in the UK, and Valve, Electronic Arts, BoltPeters, and VMC Labs in the US and Canada. The interest in this area comes from both academia and industry, sometimes enabling collaborative work. Game UX work has been featured at professional venues, such as the Game Developers Conference (GDC).Zammitto, V. (2011) The Science of Play Testing: EA's Methods for User Research. Game Developers Conference 2011

Web design

User experience evaluation has become common practice in web design, especially within organizations implementing user-centered design practices. Through user testing, the user experience is constantly evaluated throughout the whole product design life-cycle.

See also

* User experience design
* Experience modifier
* Microsoft reaction card method

References

{{Reflist

Human–computer interaction
User interfaces
Evaluation methods