User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data Analysis

Unit 7: Quantitative Methods:
Questionnaires, Biometrics and Data Analysis
Second term, January 2019 Dr. Marc Miquel Ribé
Course in User Experience
Bachelor Degree in Video Game Design and Production
Computer Engineering for Information System Management

UX
Our aim is to understand the player experience.
There are four key aspects to understanding
players and gameplay:
• User Experience: How does this make her feel
now?
• Physiological reactions: How does this make
her react?
• Recalled Experience: What does she think it
happened?
• User behaviour / interactions: What is she
doing?
Physiological reactions
User behaviour / interactions
Recalled UX (later)
Now
Quick Reminder. What are we studying in Games User Research?
We need to decide the method according to what we are studying. For each method, we have to
be very careful. Answering these two questions can be useful:
• Where and when can we bias or spoil the data?
• What are the limits of these data to reach conclusions?
Remember, we cannot know about their behaviour by just asking, or we cannot know about
the experience by just analysing behavioural data.

Our aim is to understand the player experience.
There are four key aspects to understanding
players and gameplay:
• User Experience: How does this make her feel
now?
• Physiological reactions: How does this make
her react?
• Recalled Experience: What does she think it
happened?
• User behaviour / interactions: What is she
doing?
Quick Reminder. What are we studying in Games User Research?
We need to decide the method according to what we are studying. For each method, we have to
be very careful. Answering these two questions can be useful:
• Where and when can we bias or spoil the data?
• What are the limits of these data to reach conclusions?
Remember, we cannot know about their behaviour by just asking, or we cannot know about
the experience by just analysing behavioural data.
UX
Recalled UX (later)
Now

Goal of the Unit
Learn when and how to use the methods aimed at quantifying
data from a group of users in order to draw general
conclusions.

Overview of the Lesson
7.1 Questionnaires
7.2 Biometric Methods
7.3 (Behaviour) Data Analysis
Goal of the Lesson: Learn when and how to use the methods aimed at obtaining
user’s quantitative data.

5.4 Questionnaires
Interviews and focus groups may be subject to some specific kind of bias introduced by the
interviewer and the group dynamics. To put it simply: questionnaires are a list of questions in
regards of your game. They have with different sorts of limitations but without the previous
interfering biases.
Questionnaires are also called ‘self-reported methods’ because it is the participant who
provides the data with no need of a moderator.
The questionnaire method is a research method, a tool, used to collect information from a
representative sample of a population (we aim at the target more than with an interview or
playtesting).
Differently than the interview, the survey is very good at extracting general conclusions.
Also, it is a great method for when we know what we want to know.

UX
Recalled UX (later)a) What are we studying with a questionnaire?
In a questionnaire, we only study the recalled
experience – What does she think it happened?
The recalled UX are the emotions, motivations,
beliefs, perceived usability, perceived difficulty and
mechanics, perceived fairness, among others.
5.4.1 Purpose and problems (what)
We cannot know about the real gameplay (game functioning and usability) with just the survey.
We cannot know about the user behaviour and interactions – intensity or repetition.
This is why sometimes questionnaire are conducted after the playtesting – as a complementary
but not essential or as ‘core’ as playtesting or data analytics.
We use questionnaire when we know quite clearly what we want to know.

a) What are we studying with a questionnaire?
In a questionnaire, we only study the recalled
experience – What does she think it happened?
The recalled UX are the emotions, motivations,
beliefs, perceived usability, perceived difficulty and
mechanics, perceived fairness, among others.
5.4.1 Purpose and problems (what)
We cannot know about the real gameplay (game functioning and usability) with just the survey.
We cannot know about the user behaviour and interactions – intensity or repetition.
This is why sometimes questionnaire are conducted after the playtesting – as a complementary
but not essential or as ‘core’ as playtesting or data analytics.
We use questionnaire when we know quite clearly what we want to know.
UX
Recalled UX (later)
Now

b) What kind of method it is Mainly quantitative and attitudinal
Datasource
eye tracking
data analysis
interviews & focus groups surveys
physiological data
Questionnaire gives valuable data sorted in different questions that can be closed-questions
(only few answers are available) or opened-questions (then it would also be qualitative).
playtesting/usability
think-aloud

c) Experiment structure
Purpose: A survey or questionnaire are conclusive for games.
What to ask:
Specific concept or aspect in particular with a theoretical construct. The main purpose of a
survey is to obtain useful data about a particular aspect.
The main topics that a questionnaires can be designed for can be: emotional aspects,
preferences, motivation, comprehension (cognitive), game aspects (balance, narrative, goal
clarity, etcetera.). With a questionnaire we are not going to validate behaviour or ask for
particular choices.
Research process
e.g. Motivation > Self-Determination Theory > Dimensions (Intrinsic, Extrinsic; Needs:
competence, relational, autonomy) > Indicators (Intrinsic = no rewards) > Hypothesis (the game
is dominated by intrinsic) > Method (Questionnaire) > Questions (‘You had the most fun while
maneuvering the car’> Results > Conclusions (the game is dominated by intrinsic but…).
Research Question > Theoretical Framework > Survey > Data Analysis with Statistics > Conclusions

Purpose: A survey or questionnaire can be explorative to understand the context of a user when
we are designing a new service.
What to ask:
Surveys are useful to identify:
Who your users are; What your users want; What they purchase; Where they shop;
What they own; and What they think of your brand or product.
Research process
e.g. Motivation > Self-Determination Theory > Dimensions (Intrinsic, Extrinsic; Needs:
competence, relational, autonomy) > Indicators (Intrinsic = no rewards) > Hypothesis (the game
is dominated by intrinsic) > Method (Questionnaire) > Questions (‘You had the most fun while
maneuvering the car’> Results > Conclusions (the game is dominated by intrinsic but…).
[https://uxmastery.com/better-user-research-through-surveys/]

Necessarily, the target player. It makes no sense to propose a survey to a non-target player.
We need to extract conclusions from a sample population from the target player.
Since the method is no qualitative, introducing noise in the sample would be devastating.
The sample size will be determined by the final number of players and the confidence interval
and level we want to obtain. The bigger the sample, the less error.
5.4.2 Production pipeline (when) Here! Here!
Some questionnaires are handed after the
playtesting session and are directly related to the
recent experience. Surveys are not usually for
product release.
Other questionnaires are more about the general
use of the game. It is usual to use questionnaires
after the game release in order to know better the
target player and understand the game success or
failure.
5.4.3 Participants (who)

5.4.4 Question formulation (how)
• Each question must be mapped or related to a particular concept or problem you are
trying to understand (research question).
Let us say we want to study emotional attachment or motivation (again).
Validity: A questionnaire experiment is valid if it measures what it claims to measure. For
example, a test measuring the emotional attachment to a set of characters should not ask
something else such as the UI/UX design. Another way to think about it is if I’m trying to
measure a player’s attitude of a game’s level difficulty, am I asking the correct questions to be
able to gauge that?
In order to be sure that we are asking questions about the concepts or topics we are thinking
about, we should give the questionnaire to someone to try to identify the concepts and check
the agreement.
The number of questions will determine the time to answer (it should not exceed 15-20 min.).
[http://www.gamasutra.com/view/feature/169069/finding_out_what_they_think_a_.php?print=1]
[http://www.gamasutra.com/blogs/NeferDoane/20150316/238780/How_to_Design_a_Survey_for_User_Feedba
ck.php?print=1]

General writing advices
• Use clear, concise everyday, simple language. Avoid jargon, technical terms, and acronyms.
• Refrain from using double negatives in a survey.
• Avoid double-barreled questions. A double-barreled question is a single question that asks
about more than one issue but is limited to one answer. “I found this game to be challenging
and fun.” The matters of challenge and fun are two separate issues.
• Beware of leading or loaded questions. We don’t want participants to say what we want
them to say, we want their own opinions.

Generally, the most common type of close-ended survey questions is the single-answer
multiple-choice question.
In order for close-ended questions to be comfortably answerable by the respondents, the
answers should be specific, exhaustive, and mutually exclusive.
You can apply this to ask their favorite ‘weapons’, their enemies, their avatar purchases, among
others. In a questionnaire questions must be more precise than in an interview.
For instance: if you are asking how much they spend playing. The answers should not be some,
a lot, every day. They should be ‘none’, ‘0 to 5 hours’, ‘6 to 10 hours’, ‘11 to 20 hours’, ‘more
than 20 hours’.
5.4.5 Question formats (how)

• Likert Scale: this method was
designed to 1) measure attitudes or
opinions and 2) the intensity of these
attitudes or opinions in 3) a response
format of fixed choice.
Control
Avoid asking duplicate questions in
different way (unless it is for control). For
example, using reverse scores. This asks
to rate the direction (strength of feeling)
of the statement e.g. positive to negative.
Generally, avoid questions that are
phrased negatively. But, in small doses,
counterbalance including reversed score
items. This ensures that the participant is
consistent with his ideas.
Some are five-point and some are seven-point.
Each answer in a scale has a value (1-5)
Example of Reverse Scores:
This game was fun to play.
I would recommend this game to friends.
This game was boring.
Using scales:

I was made a survey on satisfaction after a usability testing and made several mistakes. All
the answers were very positive (5/5).
What is satisfaction? “The state of satisfaction may include a variety of emotions and …
their intensity may vary according to how much you care….”
Mistakes I made:
• Be aware of biases. Their relationship with the facilitator/company.
• If you ask about satisfaction, never use 5 scale as it has not resolution enough.
Advice:
Comparing it to other experiences may be very helpful. Otherwise, what does 7 mean?
How was your experience with our company?
Using scales to understand user satisfaction with a product

• Use balanced ratings scales. Use an equal number of positive and negative options—this
relates to probability. With 4 options the natural spread would be 25% per answer, therefore
if we have more positive options than negative we would increase the chances of getting
positive feedback.
Using balanced rating scales:
With a balanced rating scale there is a greater chance of the results reflecting a participant’s true
beliefs.

If the questionnaire is aiming at understanding a very specific thing and not motivation or
emotion, you can use game elements or scenes to help ensure the player knows what you mean.
Be aware of the biases you introduce (for instance, showing just specific images and then not
others may influence the answers in further unrelated questions).
There are scales up to 10,
sometimes more distance is
not an advantage.
In order for the
questionnaire to be reliable
(this means that all the
questions represent well the
concepts). You need to keep
the same scale (either with
5, 7 or 10 levels) for the
entire questionnaire.

Order matters. Previous questions can influence further questions, especially when they are
long. You have two options. In case you are asking about a playtesting session, you can segment
the questions and introduce them like a story (beginning, development, ending), so that each
question helps remembering the recent experience. In case you are asking about a general
experience about the game (for instance, we again want to know what motivates the player),
we are going to provide very different sort of questions and shuffle them.
Anonymity. Some personalities avoid ’extremes’. Allowing anonymous answers may help.

5.4.6 Pros / Cons
Pros
• Cost-efficiency: Ease of gathering large amounts of data, practically no constraints on
geographical location and demographics, and quick results — in other words… time and
money. While a survey’s construction can take considerable effort, the player’s role in filling it
out usually takes a short amount of time.
• Consistent and reliable. You know that participants are answering to the same questions and
you can work out this data together.
• Quantifiable. You can compare different concepts and see their magnitude much more easily
(in comparison with interview) and in a large scale.
Cons.
• Individuals might be unfamiliar with or have difficulty remembering relevant information. An
additional reason for this is that individuals may feel societal pressure to respond in a manner
that will be viewed favorably by others, a social science term called social desirability bias.
• Another challenge when using survey design is that surveys have inflexible design (you can’t
change anything about the survey once you’ve started to administer it). The reasons for this
inflexibility is to maintain the validity of collected data. (for instance, it is no possible to add
follow up questions). Besides, it takes some time to create them in a proper way.
• Need a large(ish) sample to be sure that the data is representative.
• It reflects what users say instead of what users do (typical limits of subjective data).
[http://www.gamasutra.com/blogs/NeferDoane/20150316/238780/How_to_Design_a_Survey_for_User_Feed
back.php]

Case Study: Pokémon Go players’ motivation
1. Research Question: What types of motivation exist while playing Pokémon Go?
2. Concepts/Theoretical Framework: Self-determination theory, identity-based motivation, etc.
3. Methodology: Surveys (e.g. PENS), Interview, etc.

Case Study:
Research Question: What features would users want from an alarm clock app for the
metro?
Context: Alarm clock App.

5.4.7 Standardized Questionnaires
Some games user researchers create questionnaires to understand specific concepts (motivation,
engagement, etc.) and publish them so they can be used with very different players and games.
• Scott Rigby and Richard Ryan created Player Experience of Need Satisfaction (PENS), an
standardized questionnaire to examine players motivation. They also created a company
(Immersyve) to test their model with video games and give consultancy to developers.
• Their research shows that this underlying motivational energy takes the form of three basic
psychological needs: Those of competence, autonomy, and relatedness.
You can measure for these motivators in a playtest session by having players complete short
questionnaires immediately following important events in the game, such as a difficult puzzle or
a pitched battle. For example, you might ask players to rate the extent to which they agree or
disagree with these statements:
“The game kept me on my toes but did not overwhelm me.” (to measure competence)
“I felt controlled and pressured to act a certain way.” (to measure autonomy)
“I formed meaningful connections with other people.” (to measure relatedness)
With PENS, you can see the relationship between: 1) the player’s psychological
needs triggered by your game, 2) specific outcomes (e.g. Fun/enjoyment; will
buy more of developer’s game; will recommend to others).

Rigby and Ryan have demonstrated
there exist a relationship between
among the three measures of
motivation (autonomy, relational and
competence) and the player
outcomes (in enjoyment,
recommending the game….) for
specific genres of games.
For example, the PENS measures
were found to be very accurate at
predicting the likelihood that players
of an adventure or role-playing game
would purchase more games from
the same developer.
[http://www.gamasutra.com/view/feature/130155/rethinking_carrots_a_new_method_.php?print=1]
[http://immersyve.com/white-paper-the-player-experience-of-need-satisfaction-pens-2007/]
Playful Design. John Ferrara. 2012. Chapter 8. Playtesting. Evaluating Motivation: The PENS Moodel. (p. 108)

• The development of the Game Engagement Questionnaire: A measure of engagement in
video game-playing Jeanne H. Brockmyer, Christine M. Fox, Kathleen A. Curtiss, Evan
McBroom, Kimberly M. Burkhart, Jacquelyn N. Pidruzny.
• Game Experience Questionnaire (GEQ), IJsselsteijn, W.A.; de Kort, Y.A.W.; Poels, K.
Once a questionnaire is published and other scientific peers find it useful, they start using it for
their research. For instance, Chek Tien Tan et al. (2014) use GEQ to understand the relationship
between Facial Expressions and game experience. (Correlation between Facial Expressions and
the Game Experience Questionnaire).
Other questionnaires are more focused in understanding specific emotions.
IMPORTANT! Standardized and published questionnaire are often easy to find and
you can use them to obtain good feedback. You can be certain the questions and all
the details are designed in the most rigorous way.

6.1 Biometrics Methods
Biometrics is the technical term for body measurements and calculations. Biometrics methods
are those aimed at obtaining these measurements.
They may remind of you a ‘lie detector’. When utilizing lie detector machines, the body of a test
subject is hooked to several sensors capable of recording changes in a range of physiological
processes such as her heartbeat, the electrical conductivity of her skin, the frequency of her
respiration, and so forth.
[http://www.gamasutra.com/view/feature/183887/the_case_for_casual_biometrics.php?print=1]

a) What are we studying with psycho-physiological
methods?
Physiological reactions are good proxies for certain
aspects of the UX. We can really relate body
reactions to emotions and psychological states more
generally.
Through psycho-physiological methods we can detect
emotions (among many other UX aspects) that the
player is not aware of. They are produced
involuntarily.
It is not what they say about what they feel, it is what
they feel directly. Their measurement is independent
from the user talking.
6.1.1 Purposes and problems (what)
UX
Recalled UX (later)
They are based on the fact that the body gives clues on cognition and emotion.

a) What are we studying with psycho-physiological
methods?
Physiological reactions are good proxies for certain
aspects of the UX. We can really relate body
reactions to emotions and psychological states more
generally.
Through psycho-physiological methods we can detect
emotions (among many other UX aspects) that the
player is not aware of. They are produced
involuntarily.
It is not what they say about what they feel, it is what
they feel directly. Their measurement is independent
from the user talking.
They are based on the fact that the body gives clues on cognition and emotion.
UX
Recalled UX (later)
Now

b) What kind of method it is Quantitative and behavioural
Datasource
playtesting/usability eye tracking
data analysis
Interviews & focus groups surveys
physiological data
think-aloud
They answer well ‘what’ and ’how much' questions (but not why or how)

Physiological methods need to be used with a theory-driven approach. The experiment must be
designed very carefully.
1. Context/Object
2. Problem/Research Question/Hypothesis
3. Concept / Theoretical Framework
4. Methods (Methodology)
5. Measurements and test design (Methodology)
6. Results analysis
7. Conclusions
8. Reporting
Important:
• We select the concept in relation to the question.
• We select the method that can provide data useful to answer the question.
• We study the specific measures and how to interpret the data to answer the question.

Research Questions
Employing biometric experiments and methodologies to analyze a video game, you can obtain
answers to several questions related to the user experience that are crucial for its development
and commercial success. Examples of such questions include:
• Is the initial speed of our video game too stressing for our target audience?
• Did we reach a climax in emotional involvement where and when intended (this is likely to
be at the end of our free demo)?
• Does the tutorial of our video game succeed in keeping our players engaged while
empowering them to perform well?
• What are the emotions our target audience respond with to the experience of their first
Game Over?
• Is our game too stressful in a cognitive sense for them? Is it perceived as too punishing?
While asking these questions we should have clear the concepts and theoretical framework.
The next steps are choosing the method (6.1.2), studying the measurements and designing the
test (6.1.3), and finally, analyzing the data (6.1.4).

6.1.2 Methods: types of sensor (what)
• Electroencephalography (EEG): electrical brain activity measured with electrodes.
• Electromyography (EMG): recording electric potential generated by muscles. Measure of
valence (positive/negative emotion).
• Galvanic skin response (GSR): also electrodermal response (EDR) or activity (EDA),
psychogalvanic reflex (PGR), skin conductance response (SCR). Electrical resistance of the
skin. Measure of arousal (high/low).
• Electrocardiography (EKG): recording the electrical impulses and heart rate variability (HRV).
• Eye Tracking (ET): recording either the point of gaze (where one is looking) or the motion of
an eye relative to the head.
Chapter 14. Physiological Measures for Game Evaluation. Regan Mandryk. Game Usability.
Katherine Isbister and Noah Schaffer. (p. 207)

Choosing appropriate sensors depends on what concept you want to study, and what your
setting is like. Use the following as a guideline:
• Mental effort: depending on your setting, decreasing heart rate variability (HRV) or greater
pupil dilation can be used to measure increases in mental effort. Increases in jaw clenching
(through EMG sensors on the face) or brow-raising (EMG of the forehead) may also be
indicative of increased mental effort. Increased respiration rate and a decrease in the
variability of respiration rate are also associated with mental effort. Electrodes (EEG) can
also measure certain physiological aspects of mental activity.
• Positive versus negative emotions: The valence of an emotion (whether it is positive or
negative) can be measured through facial muscle analysis (EMG) over the brow (frowning)
and cheek (smiling). Some potential has been shown in the use of heart rate, irregularity of
respiration, and pupil diameter as indicators of valence.
• Arousal: increases in psychological arousal are best measured by increases in galvanic skin
response (GSR), but can also be seen in increased respiration, decreased blood volume
pulse (BVP), and increased heart rate (HR).
• Perception and attention: the time it takes to look at an element or the time spent looking
at particular elements can be indicators of perception and attention. We can measure them
with an eye tracking (ET).

Test design. Obtain the right data:
How can we obtain data that allows us to have reliable conclusions.
Once you know what you want to measure (e.g. emotion) and what sensors to use, you need to
design the test. The game user researcher usually follows an experimental psychology approach
to design the research, that is characterized by 4 features:
1. Comparing controlled conditions (always test an scenario without ’that feature’).
2. Comparing controlled participant sample (always test participants with other scenarios).
3. Representative results (right tests for measures: number of participants).
4. Counterbalanced design to remove order effects (change the order no to accumulate
undesired effects from previous interactions).
Physiological data is sensitive, variable and difficult to interpret without a high level of
experimental control.
Therefore, we are at risk to oversimplify some interpretations of physiological data and do not
keep in mind its one-to-many relationship to psychological effects.
6.1.3 Measurements: data and experiment design (how)

Test design. Some aspects to take into account:
• Sensitivity to movement: Blood volume pulse (BVP), respiration via stretch sensor, and
galvanic skin response can be sensitive to movement.
• Sensitivity to physical activity: Most physiological measures are sensitive to fluctuations in
physical activity. Be aware of this when testing users.
• Individual differences: Most physiological measures show large differences between users
and between the same user on different days or at different times of day. Use normalization
procedures to correct for fluctuations.
Physiological measures are not easy to conduct and data can be easily biased.
We need to correct deviations by applying some procedures.

What are the specific measures for each method?
Electroencephalography (EEG) measures
In terms of these frequencies, the bands of interest are usually the:
• Alpha band (8-14 hz) that reflects calm, mental work.
• Beta band (14-30 hz) that reflects focused, engaged mental work.
• Delta band (1-4 hz) that reflects sleep, relaxation and fatigue.
• Theta band (4-8 hz) that reflects emotions and sensations.
Electromyography (EMG) measures
EMG is all about detecting the activation of muscles through the use of electrodes, which are
attached to the relevant muscle (or muscles). So again, like EEG, (and like most of the measures
I am mentioning) this method relies on detecting electric current. However, unlike EEG, EMG is
a direct indication of activation in the peripheral nervous system.
• Brow (Corrugator supercilii) that register negative emotion (unpleasant valence)
• Cheeks (Zygomaticus major) that register positive emotion (pleasant valence)
• Area around the eyes (Orbicularis oculi) that are said to register expressions of enjoyment
and "genuine pleasure" (whatever that is).
[http://www.gamasutra.com/view/feature/6341/game_testing_and_research_the_.php?print=1]
6.1.3 Data analysis (how)

An eye tracker measures eye positions and eye movement. These are some of the most usual
measures and metrics that are employed:
• Fixations and gaze points
• Fixation sequences (saccades)
• Areas of Interest (AOI)
• Time to first fixation (TTFF)
• Time spent
• Pupil size (interest and attention)
Eye Tracking provides data from the gaze, in regards of perception, attention and mental processes.
Eye Tracker (ET) measures
This is a heat map created with the fixation points.

The Eye Tracking is easy to use in comparison with the presented biometrics methods.
Eye tracking can be definitive to test if something is visible enough (enemies, visual interface
elements, etc.). The time spent in an area is measure that explains well attention.
It happens often that players complaint during a post-playtesting interview something is not
visibile enough. Then during an eye tracking session it appears they can see it, although they
struggle at other parts of the interaction. Saying “it’s not visible enough is just part of the
complaint for a challenge”.
Heat map Fixations and gaze points

Case Study:
Research Question: Do all users have the same intent and pay attention to the same
places while looking at search results?
Research Question: What places do they pay more attention to?
Context: Google search results.

Case Study:
Research Question: How long does it take to users to find the Credit Card page of the
site on the home page?
Context: Google search results.

Pros
• Gives objective quantifiable data unable by any other means (it cannot be faked).
• Allows for continuous data recording without interrupting the player.
• The relationship between the player’s psychology (emotion and mental effort especially)
and some signals like heart rate and face expressions is very clear.
Cons
• The data acquisition devices are typically expensive, and sufficient attention and time should
be given to personnel training and device maintenance.
• Often invasive or intrusive.
• Problems with specificity, artifacts, inference and validity can make it difficult to interpret.
6.1.4 Pros / Cons
Eye Tracker pros (the cons are basically the same as other biometric methods)
• Its data can also be commented in a qualitative way.
• Effective metric of player attention/gaze.
• Excellent tool for interface design.
• Provides good understanding of scene interpretation.
Mixed-Methods! We want to examine the relationship between game events,
physical responses and survey results.

[https://link.springer.com/article/10.1007/s11257-017-9192-3]
Flow could be research with Eye Tracking and surveys.
• With an eye tracking we would see a sustained attention (“Focal attention is reduced to
one area and is stable along time”).
• With a survey we would see the degree of immersion they recall from their experience.
Case Study: Flow in Need for Speed 2015
1. Research Questions: Does the player experiment a Flow state of mind while playing Need for
Speed?
2. Concepts/Theoretical Framework: Flow theory, attention, physiological effects of attention.
3. Methodology: Biometric sensors: e.g. heart rate (HR), galvanic skin response (GSR),
respiration (RESP), temperature (T), blood volume pulse (BVP)
4. Results: Players experienced flow. Both data from the gameplay and the player physiological
responses could be used to understand what creates flow and then automatically generate
effective content variations to stimulate it.

Case Study: Fear in Resident Evil 7 and Agony
1. Research Question: Which game creates the strongest emotion of fear in the player, Resident
Evil 7 or Agony?
2. Concepts/Theoretical Framework: Fear, physiological effects of fear.
Interview could tell us their reasoning, how they explain fear,…Survey could tell us how much
they think they experienced fear. A survey would even be useful to see how general is the
emotion of fear among the participants.
But they are not the same things as taking physiological measurements, as we do not obtain
precise measures about their emotions. We can see better how much they experience fear.
3. Methodology: Biometric sensors: e.g. heart rate (HR), galvanic skin response (GSR),
respiration (RESP), temperature (T), blood volume pulse (BVP)

The Effectiveness of Casual Video Games in Improving Mood and Decreasing Stress. Carmen V. Russoniello,
Kevin O’Brien and Jennifer M. Parks. 2009.
[https://www.supercheats.com/articles/262/video-games-as-stress-relief/2]
Case Study: Relax in Casual Video Games
1. Research Question: Does playing casual video games such as Bejeweled 2, Bookworm
Adventures and Peggle improve mood and decrease stress?
2. Concepts/Theoretical Framework: Emotion theories.
3. Methodology: Biometric sensors: Electroencephalography (EEG), Heart Rate Variability
(HRV).
4. Results: The effects were consistent with increased mood and corroborated findings on
psychological reports. These sort of games help in releasing stress.

Why aren’t biometrics more adopted by video game developers?
Because of the “cons” and,…
1. It does not substitute playtesting, and the subjective methods (surveys, interviews) can
provide data which is useful “enough” to iterate in game design.
2. Only certain games like First-person shooters, racing cars, action-RPG, have clear
patterns in terms of subjective behavior (emotional arousal, cognitive workload). Many
other games are more fuzzy and do not put the player in these high states.
Valve’s game researcher opinion (Dr. Mike Ambinder, experimental psychologist)
"Some publishers are going down this route, but I'm not sure biometrics is the way to go. We've
had a lot of time to experience what works and what doesn't, and biometrics doesn't tend to
add a lot to the techniques we're already using. ”
“You could learn most of this stuff by just asking people. I mean, what we want to know is
whether people are having fun or not. And just asking them, or watching them play the game,
can determine that. We don't need super-precise accuracy on the emotional state. ”
Sometimes, more complex scientific methods do not add enough to justify their use.
[http://www.gamespot.com/articles/the-science-of-playtesting/1100-6323661]

6.2 (User Behaviour) Data Analysis
Telemetry n. The science and technology of automatic measurement and transmission of data
by wire, radio, or other means from remote sources, as from space vehicles, to receiving
stations for recording and analysis.
Game development telemetry – … automatic measurement and transmission of data from
game executable, build pipeline and development tools for recording, analysis and continual
improvement. What kind of data are we obtaining? Events, player small interactions,
achievements, etcetera. this is the behavioral data that we want to import from gameplay.
What is behavior? It is the user interactions along time. We can define interactions, we can
define time. By analyzing data related to the user behavior we can understand certain things
about motivation, but not completely.
Game analytics can thus be understood as the application of analytics to game development
and research (El-Nasr, Drachen and Canossa, 2013, p.5).
[http://www.gameanalytics.com/blog/what-is-game-telemetry.html]

a) What are we studying with game data
analysis?
Playtesting can shred light into ”what” and
”why”, but with game analytics we can
understand “how much” something happens.
It gives us a wider perspective, in order to
evaluate if the what happens very often and
what is extensive to all players.
Besides, we can ‘cross data’ with economical
data (revenue, conversions, etcetera.). But this
is not the scope of our study in UX.
UX
Recalled UX (later)
Data analytics is used to study user behaviour and monetization.

a) What are we studying with game data
analysis?
Playtesting can shred light into ”what” and
”why”, but with game analytics we can
understand “how much” something happens.
It gives us a wider perspective, in order to
evaluate if the what happens very often and
what is extensive to all players.
Besides, we can ‘cross data’ with economical
data (revenue, conversions, etcetera.). But this
is not the scope of our study in UX.
Data analytics is used to study user behaviour and monetization.
UX
Recalled UX (later)
Now

IMPORTANT: Once you have logs, you will never ask
again about interaction when there is limited amount of
time in an interview or in a survey.
Behavioral data is unbiased data
Which gives you the clearest picture of your game? Surveys or data logs?
Data science is the set of methods and practices to extract knowledge from data.
It mainly provides analysis and visualization. We apply data science to games.
Behavioral data does not depend on the ‘observer’ or
the player (unless there are coding bugs). It comes
directly from the game functioning. Like biometrics, it
cannot be faked by the player.

b) What kind of method it is Quantitative and behavioural
It answers well some ‘how much’ questionsDatasource
playtesting eye tracking
data analysis
Interviews & focus groups surveys
physiological data
playtesting
think-aloud

Theory-driven means that the entire research process is based on a theory which gives a model of
interpretation of the reality, then it allows you to find indicators, to create hypothesis, to run a
test and finally obtain results and conclusions. This is the scientific method explained in previous
classes. With theory-driven we want to prove a hypothesis right (or wrong) and still want an
explanation.
Research Question > Concept / Theoretical Framework > Methods > Data Analysis > Conclusions
Instead, data-driven strategies rely on solely pay attention to the data without having a strong
theory behind. In this second case, it is possible to gather data without a theory in mind, and then
‘work out’ the data until finding interesting insights. I have this data… What can I do with it?
Perhaps, later, we can find a theory which helps us at understanding the data.
Data analysis (telemetry) allows data-driven research while biometrics and surveys do not.

d) Most common concepts we study with data analytics
The most usual specific purposes to use game analytics are to balance economy, to
understand challenge level, to study game mode preferences, to study motivation and
participation, to catch cheaters.
We cannot study how users understand narrative, whether they find the controls or the user
interface usable, or the emotions they experience. Just the behavioural part.
• If you know where your players are getting
stuck, you can change the difficulty so more
players can continue playing.
• If you know where players die too easily,
you can redesign the level to prevent that
from happening.
• If you know that some game items are too
expensive, you can change the prices so
more players can afford them.
Analytics may give you a hint on why this is probably too hard! (So many deaths).

Game telemetry data can be thought of as the raw units of data that are derived remotely from
somewhere, for example an installed client submitting data about how a user interacts with a
game, transaction data from an online payment system or bug fix rates. In the case of user
behavior data, code embedded in the game client transmits data to a collection server; or the
data is collected from game servers (as used in e.g. online multi-player games like Fragile
Alliance, Quake and Battlefield) (Derosa, 2007; Kim et al., 2008; Canossa and Drachen, 2009).
6.2.2 Production pipeline (when and where)
So… it can be from the game developer
headquarters or from anywhere else in the world.
We are here Production and post-production

6.2.3 Data acquisition and pre-processing (how)
Game telemetry happens in different
phases: the first one is attribute definition.
In the following sections we are going to
see more about each phase.
1.
2.
3.
4.
5.
6.
7.
8.
An Introduction to Gameplay Data Visualization. Günter Wallner and Simone Kriglstein. Game Research
Methods. 2015.

1. Attribute definition
The essence is that telemetry is measures of the attribute of objects – the latter which
should be understood broadly to include people and processes.
For example, the location of a player character as it navigates a 3D environment. In this case
the location is the attribute, the player character the object.
In order to work with telemetry data, the attribute data needs to be operationalized. This
means deciding a way of expressing the attribute data: 0-1, numerical, etc.
Raw telemetry data can be stored in databases.
Game metrics are, in essence, interpretable measures of something, as long as this
something is related to games.
Metrics can be directly attributes or created from attributes (for instance, a ratio, the
number of events in a determined time or place).

1) General attributes: The attributes that are shared for users (players) across all games. These
form the core metrics which can always be collected, for any computer game, e.g. when a user
starts playing a game, stops playing, a userID, etc.
2) Core mechanics/design attributes: The essential attributes related to the core of the
gameplay and mechanics of the game. For example attributes related to time spent playing,
number of opponents killed, etc. Defining the core mechanics attributes should be based
directly on the key gameplay mechanics of the game, and provide information that allows
inferences to be made about the user experience. For example, whether players are
progressing as planned, if flow is sustained, death ratios, level completions, point scores, etc.
3) Core business attributes: The essential attributes related to the core of the business model
(e.g. F2P) of the company. For example, logging every time a user purchases a virtual item,
establishes a friend connection in-game, country of origin, recommends the game to a
Facebook friend, attributes related to retention, virality and churn, etc.
These other attributes
can be regarding
business,
performance, errors…
We are interested in
these two! (1 and 2)

2. Data acquisition
During this second step, incoming telemetry data are transformed and loaded into a database
structure, from where they are accessible for analysis. Additionally, data are cleaned and
otherwise made ready for analysis.
1. Start at the accounts database (in case of an external database provider). This will be the
first step to economy, since the accounts database has the ID of every record that you want
information about. Then, you can store the data. Once it is in, you can validate it.
2. Validate which data is relevant and clean. This eliminates garbage as soon as possible, so
that you are not storing or analyzing unusable data. Starting at the accounts database,
exclude unregistered accounts or administration accounts. For example, exclude test and
admin characters that have artificial attributes. For each valid character in an account, query
for activity in the log database. If the character has not been active during the previous
week, then its record contains no player performance information.
3. Backup valid user, log, and accounts records into an archive database. This will be a useful
warehouse that you may return to in the future to mine for data you have not considered
yet. Treat this backup preciously; if you were an archaeologist, this would be your find; if you
were a detective, this would be your forensic sample.

The data acquisition comprises the different computing systems and processes dedicated
to obtain and storage the data.
• The code installed in the back-end of the game to store the .
• The communications at network level (TCP-UDP)
• The database where the data is stored (MySQL, etc.)
This is the usual network architecture to do data analysis
[https://www.raywenderlich.com/7559/game-analytics-101]
System Architecture

The transmission of a piece of information via a telemetry system – irrespective of whether this is in the
context of user, process or performance measures – in games can occur in three fundamental ways:
1) Event: A pre-specified event occurs, for example, a user starts a game, a designer submits a bug fix
request, a unit of a game is sold, a player fires a weapon, buys an item, etc. – any action initiated by a
person or system forms an event. Event-based telemetry is based on tracking such actions and
transmitting this information to a collection server.
2) Frequency: Rather than being triggered by the occurrence of a specific event, information can be
recorded following a specific frequency. For example, when tracking the trajectory of player avatars
through virtual environments, we can record the location of the avatar once per second, as a
compromise between precision and bandwidth constraints. Frequency-based recording of telemetry is
generally used when the attribute of the object being tracked is always present, e.g., a player character
in an MMORPG always has a position in the world when playing.
3) Initiated: Sometimes the game analyst wants to enable and disable the tracking of a specific attribute,
rather than having a telemetry system autonomously submitting tracked information based on some
pre-defined command. For example, it may not be necessary to record player avatar trajectories all the
time, but only when updates or patches are pushed to the users. Having the ability to turn on and off
recording of specific attributes can be useful in these situations. There are different strategies available
for the recording, transmission and storage of game telemetry.
Chapter 12: Game Analytics – The Basics. Drachen et al. Game Analytics (Book).
Tracking Strategies
Strategies are important in order not to saturate the game performance and database.

[http://www.gamasutra.com/view/feature/2816/better_game_design_through_data_.php?print=1]
Player behavior is a function of the day of the week.
Depending on the target, it will make more sense to track a specific period of time.

Valve has a platform for recording gameplay metrics: Kills, Deaths, Hero Selection, In-
Game Purchases, Matchmaking wait times, Bullet trajectories, Friends in Party, Low-
Priority Penalties, etcetera.
Data sent at relevant intervals: Daily, Monthly, Lifetime Rollups, Views, Aggregations.
This is an important point: it is not useful to send data at all times. At the same time, it is
necessary to collect data for different periods of time (days, weeks, months). Take into
account the circadian patterns, holidays, etc.
Valve and its Data Collection OGS (Operational Game Stats)
[https://www.youtube.com/watch?v=HQwL6zh7AgA&list=PLckFgM6dUP2hc4iy-IdKFtqR9TeZWMPjm]

To summarize, many alternate methods can do this. Here is a simple method that economizes
storage space and reduces mining computation. This preprocess has five general steps:
1. Take a snapshot of the database.
2. Validate that the data is clean and appropriate for analysis.
3. Integrate the data into a central archive.
4. Reduce the data down to just the fields you need.
5. Transform the reduced data into a form that is easy to analyze for player performance.
We split attribute definition, data acquisition and metrics development because… we can
always create better more developed metrics. If we obtain the data and process the metric at
the same time, we can never go back to the original data.
Its best to obtain the attributes simple and then process more complex metrics.
Metrics development
3. Data pre-processing

Engagement metrics. This are usually created from general attributes which can be found in
all games. They are also very usual in websites to measure user engagement.
These are usually about the session duration, the number of return (retention or loyalty).
They usually take into account variables related to time.
4. Metrics development
Up to this point the discussion about user attributes has been at a fairly abstract level,
because it is impossible to develop classes of which user metrics it makes sense to develop
for all types of games.
FPS, TPS, Racing, Adventure Games, Arcade, Beat ’em up, Family games, Fitness games,
Music games, Platformer games, RPG, Simulation, Sports Games, Strategy Games.
Exercise! Let’s play with metrics

• FPS: Useful gameplay metrics: Weapon use, trajectory, item/asset use, character/kit choice,
level/map choice, loss/win [quota], heatmaps, team scores, map lethality, map balance,
vehicle use metrics, strategic point captures/losses, jumps, crouches, special moves, object
activation. AI-enemy damage inflicted + trajectory. Possibly even projectile tracking.
• TPS: Useful gameplay metrics: as for FPS + camera angle, character orientation.
• Racing: Useful gameplay metrics: Track choice, vehicle choice, vehicle performance, win/loss
ratio per track and vehicle, completion times, completion ratio per track and player, upgrades
[if possible], color scheme [if possible], hits, avg. speed different types of tracks/track shapes.
• Adventure games: Useful gameplay metrics: story progression [e.g. node based], NPC
interaction, trajectory, puzzle completion, character progression, character item use, world
item use, AI-enemy performance, damage taken and received + source (player, mob).
• Advance: Useful gameplay metrics: trajectory, powerup usage, special ability usage, session
length, stages completed, points reached, unlocks, opponent type damage dealt/received,
player damage dealt/received [as applicable].
• Beat’em up: Useful gameplay metrics: Character selection, ability use, combo use, damage
dealt, damage received (per ability, character etc.), weapon usage, arena choice, win/loss
ratio as a feature of character, player skill profiles.

• Family games: Useful gameplay metrics: varies substantially – subgame selection,
character/avatar selections, game mode used, in-game selections, asset use, number of
players, etc. form some of the possibilities
• Fitness games: Useful gameplay metrics: session length, calories burned, exercises chosen,
match between exercises shown and player actions, player accuracy in performing exercises,
total playtime over X days, player hardware/exercise equipment [usually registered], player
demographics [usually entered during profile creation], music tracks selected, backgrounds
selected, avatar selection, powerups/content unlocked [common feature], total duration of
play per user.
• Music games: Useful gameplay metrics: Points scored, song/track chosen, match with
rythm/auditory mechanics, difficulty setting, track vs. difficulty, track vs. errors, track vs.
choices.
• Platformer games: Useful gameplay metrics: jumping, progression speed, items collected,
powerups/abilities used, AI-enemy performance, damage taken + sources of damage
• RPGs: Useful gameplay metrics: character progression, quest completions, quest time to
complete, asset use (resources), character ability/item use [including context of use], combat
statistics, AI-enemy performance, story progression [including choices], NPC interactions [e.g.
communication], ability/item performance, damage taken + sources of damage, cutscene
viewed/skipped, items collected [including spatial info].

• Sports games: Useful gameplay metrics: match types, win/loss ratios, team selection, color
schemes, country chosen, management decisions [if game includes management aspects],
in-match events [e.g. goal scored, fouls, tackles, length of hit], item use [e.g. club type],
heatmap [density of player time spent on sections of the field], team setup/strategy, player
[in-game] selection, player commands to team/team members.
• Strategy games: Useful gameplay metrics: all features related to player strategy and control.
Generally two types of things players can build: building and units. Selections and order of
selection are crucial metrics. Commands given to units, upgrades purchased, trajectory,
win/loss ratio, team/race/color selection, maps used, map settings, match/game settings
(usually strategy games have some settings that affect the core mechanics).
Race/aspect/team chosen, time spent on building tasks vs. unit tasks.
How many metrics did you match?

We are at the data analysis and
evaluation phase and we want to
extract some interesting conclusions
from our data.
During this phases, cases and stored
metrics are selected as required by
the analysis in question.
Some basic operations are:
• To identify or classify
• To compare
• To relate
6.2.4 Data analysis (how)
Methods. 2015.
1.
2.
3.
4.
5.
6.
7.
8.

Player Modeling using Self-Organization in Tomb Raider: Underworld. Anders Drachen, Alessandro
Canossa and Georgios N. Yannakakis. (PAPER)
Modelling the entire population of players can
give a good idea in order to understand if the
game is balanced for “the most important type of
player”.
Case Study: Identifying Tomb Raider players
1. Research Question: Are there distinct player profiles who prefer
each of the different mechanics and goals provided in the game
Tomb Raider Underworld?
2. Concepts/Theoretical Framework: Bartle’s Taxonomy, game design
concepts, MMOO theory,…
3. Methodology: Data Analysis (neural networks). Metrics: causes of
death (opponent, environment, falling), number of death,
completion time, help-on-demand. In this study we identify players.
4. Results:
They identified the player types with cluster
analysis. The algorithms trained on the data reveals
four clusters of playing behavior — labeled as
Veterans, Solvers, Pacifists and Runners.

Case Study: Enabling cooperation in Left 4 Dead
1. Research Question: Does introducing a GUI marker make player cooperation more effective
in Left 4 Dead?
Problem: players letting teammates die.
Hypothesis: Give better visual cues to teammate location will increase cooperation and
reduce team death rate.
2. Concepts/Theoretical Framework: GUI design, usability, perception.
3. Methodology: Data Analysis (metric: high death rates), Surveys, Q&As. Setting “No Mercy –
The apartments”. In this study we compare metrics in different scenarios.
4. Results:
Death decreased a 40%
Survey ratings of enjoyment/cooperation increased

Case Study: Improve Player Communication in DOTA 2
1. Research Question: Does introducing alert messages and automatic bans in DOTA 2 decrease
the level of negative communication?
Hypothesis: Automating communication bans will reduce negativity in-game
Iterative (future): Will this work in Team Fortress 2? Do these systems scale?
2. Concepts/Theoretical Framework: Theory-Driven: Operant conditioning. No feedback loop
to punish negativity.
3. Methodology: Data Analysis, Chat, reports, forums, emails, quitting.
Measurements: Chat, reports, ban rates, recidivism. In this study we relate.
4. Results:
35% fewer negative words used in chat • 32% fewer communication reports • 1% of active player
base is currently banned • 61% of banned players only receive one ban.

Case Study: CS:GO weapon selection
1. Research Question: Does a wider selection of weapons increase a longer gameplay in CS:GO
multiplayer?
Iterative: Inform future design choices.
Hypothesis: M4A4 usage is high; few choices in late-game. Creating a balanced alternative
weapon will increase player choice and playtime.
2. Concepts/Theoretical Framework: Game design, balance theory (greater tactical choice ->
Player retention).
3. Methodology: Data Analysis (purchase rates, playtime). In this study we relate.
4. Results:
~ 50/50 split between new and old favorites • Increase in playtime.
Does weapon variety increase player retention? Still to answer.

Visualizations are representations of
data to perceive, use, and
communicate information.
In context of gameplay data analysis,
the interest to use and develop
visualization techniques increased in
the last years among industry
professionals and researchers. Visual
representations of gameplay can
support game developers and
designers to analyze recorded player
behavior to, for example, identify
interaction or design problems or to
understand the effects of design
decisions.
6.2.5 Reporting and data visualization
Methods. 2015.
[http://www.gamasutra.com/view/feature/170332/?print=1]
1.
2.
3.
4.
5.
6.
7.
8.

Methods. 2015.
[http://www.gamasutra.com/view/feature/170332/?print=1]
The power of visualization: on the left is where people are standing when they make kills with a
weapon and on the right is deaths by this weapon in Halo Reach. With just a basic knowledge of
FPS games, you can still probably work exactly what kind of weapon this is and where the
elevated and the open spaces are in the level.
They are less precise than statistical analysis
but sometimes more helpful. Easier to
understand and find interesting insights.

Charts are pictorial representations of information. Charts exist in a variety of forms, like bar
charts, pie charts, or scatter plots to name but a few with each of them having different
advantages and disadvantages.
Four general types of data visualization
Pie-charts show which types of towers have been built on the different building lots in Team Fortress.
The radius of the pie-chart is proportional to the number of towers built (Kayali, et al., 2014).
Advantage: they can summarize variables and display trends more easily.
Disadvantage: they can lead to false conclusions if the chart is inappropriate.
Charts; HeatMaps; Movement Visualizations; Node-Link Representation

Heatmaps are used to visualize aggregated data from huge data sets but can also be used to
provide players with individualized visual feedback for the purpose of post-gameplay analysis.
Heatmap of death locations on the Team Fortress 2 map Goldrush.
[http://www.gamasutra.com/view/news/125213/Opinion_Balance_and_Flow_Maps.php]
Advantage: they can show spatial patterns more easily.
Disadvantage: only one variable can be shown at a time and a third dimension is lost.

[http://www.gamasutra.com/view/news/125213/Opinion_Balance_and_Flow_Maps.php]
It is possible to create several versions of the same map with subtracted metrics
(player kills – player deaths) to obtain an idea of balance.
A balance heat map can show us perfect spots
from where people kill and are not killed.

Movement visualizations can help you understand how players actually move around in a game can
thus provide valuable information for level design.
It can be interesting to use colors in order to clear
what is the achievements the player finally
obtains or how he died.
Advantage: they show exploration patterns.
Disadvantage: a lot of data can be difficult to
visualize.

Node-link approaches provide an intuitive way to visualize the relational structure of data items.
Left: Player movement between regions, cities, and battlegrounds on the World of Warcraft
continent Outland. Right: Corresponding matrix view with cells colored according to the number
of players moving from one area to another.
Advantage: suitable for multidimensional or abstract data.
Disadvantage: dense graphs suffer from visual clutter, layout can look confusing without something
to orient the data.

Case Study: Bioware’s Star Wars Area Balancing
1. Research Question: Are there unbalanced and conflicted areas in Star Wars: The Old Republic
that drive players to a bad experience?
Content interation can help at balancing.
1. Concepts/Theoretical Framework: Game balance theories
2. Methodology: Communication Channels, Data Visualization, Content Tracking.

The interesting point is that BIOWARE not only has good metrics but every item in the game
(asset) tracked. So they can cross even more data.
Georg Zoeller MMO Content Iteration
[http://twvideo01.ubm-us.net/o1/vault/gdconline11/Georg_Zoeller_Rapid_MMO_Content_Iteration.pdf]
[http://gdc.gulbsoft.org/2011-gdc-online-talk]

“Almost all actionable content feedback is more useful when you look at from a spatial or
temporal perspective. In order to create an efficient iteration process, we need to look at all
three elements together.” Zoeller, BIOWARE
With excellent tools

We want to make it possible for people in the trenches to analyze and suggest course of action to their
leads. We also want them to be able to spot mistakes on their own – something the tool can help with by
highlighting common mistakes (2 strong enemies on a single encounter, etc.)

Pros
• Objective data that can be collected remotely and discretely.
• It can be used to see trends and provide data that supports other methods.
• It allows for continuous data recording without interrupting the player
• You can take robust conclusions from an entire population of players.
• You can start research from the data without the need for a theory.
• They are very useful for post-production (mobile games) or multiplayer games (MMORG).
Cons
• Time and resources-consuming.
• It requires good professionals with programming and statistics skills.
• Needs large sample sizes to get meaningful data. And at the same time, you can get too
much data.
• No subjective feedback, so you can never really tell what is going if you have no other
source of data. You can make inferences about usability or motivation, but not with the
same certainty that other data would provide you.
• It does not clearly substitute any other methodology (playtesting, survey, interviews).
• Behavioral data is not good at explaining ’why’ or giving new ideas.
6.2.6 Pros and cons

Hodent (2017) reminds us of some classical statistical fallacies. There are things to keep in mind:
• Sample representativeness. You cannot generalize from a small sample.
• Is the result statistically significant? Average are not enough. You need statistical tests.
• Correlation is not causation. Be cautious. You do not know the nature of the relationship
with a correlation. Perhaps it is a coincidence and there is no pattern.
• Data are not information and information is not insight. Tons of data is not the answer if we
have no questions or we are not good at interpreting it.
• Bad data are worse than no data. If you have a bug in the telemetry it can be fatal.
• Data analysis is good at telling what is going on but not why. You should complement it with
other methods.
6.2.6 Statistical fallacies and limitations

[http://www.gamasutra.com/view/feature/5827/starcraft_ii_building_on_the_beta.php?print=1]
Analyzing behavioral data without a feel on
playtesting can be misleading.
If we look at the stats and we say,
"This doesn't actually back anything
we're experiencing online," I'm very
suspicious of that number.
We look at another source and say,
"You know what? What they're saying
online matches my play experience,
and it matches the stats. This seems
real. Let's talk about what some
possible fixes can be."
Data-driven assumptions wrong: Starcraft II carriers
Design director Dustin Browder warns that caution is required when analysing the data.
• “With unit stats, I can tell you that, for example, in a Protoss versus Terran game, 12 percent
of the time the Protoss build carriers. And when they build carriers, they win 70 percent of the
time. You could say, "That must mean carriers are overpowered!”. Not, it just means that you
get towards the end of the game. If they have extra resources to waste, they’re going to win
anyway.”

Investigate suspicious player performance starting at the top.
Data-driven assumptions right: Preventing Cheaters
• A cheater in a MMOG does not just cheat himself. He performs an injustice to all honest
players. Cheating short-circuits gameplay, so it achieves exceptionally high performance.

You can only check Google Analytics in post-production.
Fidelity
Frustration
Classic usability
Conversions
tasques més freqüents, cerques més freqüents...
rati de conversió.
rati de compra.
Search analytics
6.2.7 Web: Google Analytics

Case Study:
Research Question: Which wording is best in order to stimulate converstion?
Context: e-commerce website.

Key Questions and Concepts (TakeAways)
• Quantitative data is not possible to bias with the gathering procedures.
However, you need expertise in obtaining and analysing them.
• Biometrics may be useful to understand very specific aspects of user
experience, but playtesting and interviews may be enough just to provide
insight to game design.
• Data analytics is fundamental to study the characteristics of the players
that end up playing your game. It provides valuable data but it also needs
to be contextualized with qualitative data.

References and Bibliography
• All the references provided in the Powerpoint are valuable.
Books
• Game Analytics, Maximizing the Value of Player Data. El-Nasr et al. 2013.
• Chapter 12: Game Analytics – The Basics. Anders Drachen, Magy Seif El-Nasr, Alessandro Canossa. Game
Analytics (Book).
• Game Usability: Advancing the player experience. Isbister, Katherine, and Noah Schaffer. CRC Press. 2015.
• Game Research methods: An overview. Lankoski, P., & Björk, S. 2015.
• Games User Research: A Case Study Approach. Miguel Angel Garcia-Ruiz. AK Peters/CRC Press. 2016.
• Others
• Methods for Game User Research - Studying Player Behavior to Enhance Game Design. Heather Desurvire
and Magy Seif El-Nasr. 2013. (PAPER)
• Articles available in the site Gamasutra [gamasutra.com]
• Blog gameanalytics
– [https://andersdrachen.com/2013/10/31/10-great-reads-on-gamef-analytics/]
– [https://andersdrachen.com/category/game-user-research]
• CASE STUDY: "Game Analytics" book. Chapter 16: Better Game Experience Through Game Metrics: A Rally
Videogame Case Study. Pietro Guardini and Paolo Maninetti. (PAPER)
• CASE STUDY: Game Metrics for Evaluating Social In-game Behavior and Interaction in Multiplayer Games.
Katharina Emmerich. (PAPER)
• The Game Life-Cycle and Game Analytics: What Metrics Matter When? | Mark GAZECKI
[https://www.youtube.com/watch?v=C5lx4L0iJQI]
All images used in these slides belong to the cited sources.

• Nacke, L. E. (2013). An introduction to physiological player metrics for evaluating games. In Game
Analytics (pp. 585-619). Springer, London.
• “Beyond Thunderdome: Debating the effectiveness of different user-research techniques”
[https://vimeo.com/26733185]
• [http://www.gamasutra.com/blogs/TrevorMcCalmont/20130208/186075/5_Common_Pitfalls_for_Mobil
e_Game_Analytics.php]
• Game Usability: Advancing the player experience. Isbister, Katherine, and Noah Schaffer. CRC Press. 2015.
• Game Research methods: An overview. Lankoski, P., & Björk, S. 2015.
• Games User Research: A Case Study Approach. Miguel Angel Garcia-Ruiz. AK Peters/CRC Press. 2016.
• Playful Design. John Ferrara. Rosenfeld Media, 2012.
• The Art of Game Design: A Book Of Lenses. Jesse Schell. Carnegie Mellon University. 2008.
• King, R., Churchill, E. F., & Tan, C. (2017). Designing with Data: Improving the User Experience with A/B
Testing. " O'Reilly Media, Inc.".
All images used in these slides belong to the cited sources.

User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data Analysis

More Related Content

Similar to User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data Analysis

More from Marc Miquel

Recently uploaded

User Experience 7: Quantitative Methods, Questionnaires, Biometrics and Data Analysis