4.6.1.2 Create a Study Design Document

• A study design document should capture the decisions you make when
  planning your study.

4.6.1.3 Make Scripts for Participant Visits

• If you are interacting with participants, create a script document for each visit.

4.6.1.4 Pilot Your Study

• In a pilot study, you run a group of people through the entire study from
  the beginning to end as if they were real participants.

4.6.2.1 Make Your Technology Robust Enough

• The “enough” part of “robust enough” is very important in managing the
  effort involved in the study

4.6.2.2 Consider Other Evaluation Methods

• Before taking the large step of deploying your technology in the wild,
  consider other evaluation methods to identifying as many usability problems
  as possible. In heuristic evaluation, developed by Nielsen and Molich
  (1990), a set of evaluators (which could be you and your colleagues) uses a
  small set of heuristics to critique your technology and identify problems
• Laboratory studies before your field study
  can also be very valuable to ensure that your technology is usable.

4.6.2.4 Get Reassuring Feedback

• Once your technology has gone into the field, look for means to reassure
  yourself it is working as you expect
• As mentioned previously, if your technology is not logging data to a central server, consider having it send you periodic “everything’s fine” messages so you can detect problems as soon as possible.

4.6.2.5 Negative Results

• Do not plan a study
  that relies on adoption and usage as the only dependent variable, because
  you will be in trouble if people do not adopt your technology.

4.6.4.3 Do Not Make Inappropriate Claims

• A limitations section in a paper or
  presentation that acknowledges potential limitations (e.g., a small number
  of participants from a limited geographic region) of the study helps make
  clear to the audience that you are not making inappropriate claims

• In a control condition, you measure the dependent variables for a certain
  period before you introduce the technology (e.g., logging for a week a behavior
  that you think might change), then introduce your technology and measure
  the dependent variables again
• However, collecting control data is not appropriate for all ubicomp studies,
  because your prototype may afford a behavior that was impossible without
  it and thus there is no meaningful control condition to compare against.
• For example, if you wanted to give the location-based mobile application to
  people that had never used a mobile phone before, you could not compare
  against previous use of mobile phones, but you might try to collect data
  about how often the participant communicated using landline phones or
  other communication methods to compare against.
• In addition to deciding on your study method, if you are introducing
  a new technology in your study, there are a number of pragmatic
  considerations
  a. Will participants use the technology as they • wish or to complete
  specific tasks?
  b. Will you give the participant technology to use or augment the technology
  the participant already owns?
  c. Should you simulate any part of the participant’s experience?

• In field studies, logging is often the main method for collecting quantitative
  data about usage, either of existing technology or your novel technology.
  When logging data, your prototype typically writes information to
  a data file when things occur that you want to know about.
  

• Surveys are often used to gather data before a field study begins (presurvey),
  after any changes of condition in a between-subjects study (postcondition),
  and at the end of the study (postsurvey).

• In ESM, participants are
  asked to fill out short questionnaires at various points throughout their day,
  asking about their experience at that time.
• ESM allows the researcher to collect
  qualitative data throughout the study, which has advantages over asking
  participants later to try to recall what they were thinking or feeling, or why
  they took some action
• Participants can be asked to complete a survey either
  randomly throughout the day, at scheduled times, or based on an event.
• Although it
  is most often used to gather qualitative data, you can also use ESM to gather
  quantitative data based on events, for example, recording the location of a
  participant every time he or she answers a call on their mobile phone.

• Similar in spirit to ESM, some studies gather data by asking participants to
  record information about what they do, typically referred to as a “diary.”
• This method is frequently used when participants are making diary entries
  about something that would not be possible to sense using an ESM tool, thereby rendering event-based ESM inappropriate.
• Many of the considerations for diary studies are similar to those for ESM, such as what you will ask your participant to record in each diary entry.
• However, for diary studies there are typically greater concerns about participation, because participants are typically not carrying a device that interrupts them as
  in an ESM study.
• Another option is asking participant to retrospectively construct a diary, as the Phone Proximity study had participants do at the weekly interview for the previous day using the Day Reconstruction Method

• During field studies, researchers frequently conduct “semistructured
  interviews.”
• Retrospective interviews using video can be
  a valuable method to use for asking participants about situations in which
  they cannot be interrupted (e.g., playing basketball, performing surgery).

• What type of study is it? Proof-of-concept • studies may be on the
  shorter side if less time is needed to prove the feasibility of the prototype.
• Studies of experience using a prototype are usually
  longer because the study is the contribution, whereas studies
  of current behavior vary widely.
• Do you expect novelty effects to be an issue? Often, when using new
  technology, people start out very enthusiastically using it and then
  decrease their usage. Unfortunately, there is no guarantee about how
  long novelty effects last. If you are worried about novelty effects, try
  to make your study as long as possible and be wary of basing too
  many of your findings on usage from the beginning of the study
  

• Participants should receive a consent form at the beginning of the study
  to review and sign to signify that they have consented to participate

• Identifying the participants you would like to recruit for your field study
  depends on the research goals of your study
• It is also best to have participants
  who are not involved in any way with your research. This reduces
  the chances that they are biased by knowledge they might have of your
  study or goals. Finally, recruiting different types of participants and comparing
  between them is a common type of independent variable
• recognize that your research question will help you decide how
  to rank the importance of different aspects of your participant profile.
  Depending on what is important for your study, you may be forced to make
  trade-offs in other criteria.

• Are there any conditions in your study (• e.g., between or within
  subjects)?
• What claims are you trying to make? Is this a proof-of-concept
  study?
• Plan for participants to drop out.
• Time to recruit participants

• However, as researchers in
  that study noted, when you do not compensate your participants you need
  to consider bias.
• Will the compensation method affect the • data collected?

• The statistics that are appropriate
  to use depend on how a variable was measured, referred to as its
  level of measurement
• The three common levels of measurement for field study variables
  are described below
  i. Nominal Variables where the possible answers represent unordered
  categories are referred to as nominal, or sometimes categorical. For
  nominal variables, you can only report the frequency that each category
  occurred. For example, gender is a nominal variable where the count of
  responses can be reported (e.g., Phone Proximity study had 10 male and
  10 female participants), but there is no concept of ordering between the
  response categories
  
  

ii. Ordinal Variables measured on an ordinal scale represent a rank
order preference without a precise numeric difference between different
categories. For example, a survey question with five possible responses of
daily, weekly, monthly, and almost never, is measured on an ordinal scale,
because the response options can be ordered from more to less frequent,
but not added or subtracted. For ordinal variables, both the frequency that
each category occurred and the median value can be reported.

• Answers to Likert scale questions on a survey are the most common
  example of ordinal variables collected during a field study. You can compare
  whether different participant’s answers are more positive or less positive
  than another, but they cannot be added or subtracted
  
  

iii. Interval For variables measured on an interval scale, the difference
between any two values is numerically meaningful. Interval variables
can be added and subtracted—for example, a person’s age in years,
the number of times someone performed a particular action, how long
an action took, or the number of ESM surveys a participant answered.
Descriptive statistics valid for interval data include sum, mean, and
median.

• It is important to examine interval data for outliers. Outliers affect the
  mean, so always report the standard deviation if you report the mean value
  for a variable.

• Once you have computed descriptive statistics for a variable, one type of
  inferential statistics, significance tests, allow you to determine whether the
  results found in your sample of participants are statistically significant or
  might be due to sampling errors.
• The use of inferential statistics in analyzing
  field study data is rare since the small number of participants typically
  feasible to have in a field study makes it difficult to collect enough data for
  many statistical tests to be appropriate
• To conduct a significance test comparing descriptive statistics, you first
  determine the variable you wish to compare and the appropriate groups of
  participants or different conditions to compare between.
• For example, an independent samples
  t-test is appropriate to use when comparing the mean of a variable
  with a normal distribution between two groups, whereas analysis of variance
  (ANOVA) tests are used for comparing across more than two groups.
• However, many of the nonparametric
  equivalents (e.g., Mann-Whitney U, Kruskal-Wallis) that do not assume
  a variable has a normal distribution, may be more appropriate for field
  study data since they make fewer assumptions about the data that have
  been collected.
• Regardless of what statistical test you use, significance tests start with
  the assumption that there is no difference between the groups for the variable
  being examined (referred to as the null hypothesis). If a difference
  is observed (e.g., the means or medians are different), there are two possibilities:
  there is a difference between the groups or that there is sampling
  error in the data.
• The p value indicates how likely it is that the data might
  be wrong. Researchers often use a cutoff of either p < 0.01 or p < 0.05 to
  determine if the test results are statistically significant. If p < 0.01, there is
  a 99% chance that the data collected represent a real difference between
  the groups rather than a sampling error (or a 95% chance for p < 0.05).

• Strauss and Corbin (1998, p. 3) broadly define coding as “the analytic
  processes through which data are fractured, conceptualized and turned
  into theory.”
• However, the simplest coding techniques consist of closely
  examining your data and counting the number of times a concept or
  theme reoccurs, essentially turning qualitative data into quantitative data.
• Depending on your study, it may be appropriate to have one person code
  the data. However, multiple coders, sometimes referred to as raters, are
  often used if there is a large amount of data to code. When multiple raters
  code, it is necessary to check for interrater reliability, agreement between
  the raters, to make sure different people are coding the data consistently.
• More typically, multiple
  raters each code the same subset of data (in addition to mutually exclusive
  subsets), and then a test such as Cohen’s kappa is used to report interrater
  reliability on the overlapping set of data and show that the raters are coding
  consistently.

• Two common methods used are affinity diagramming
  and grounded theory.
• Based on the affinity process introduced by Kawakita, Beyer and
  Holtzblatt (1998) created an affinity diagramming process, as part of their
  Contextual Design process to develop user-centered systems. Their affinity
  diagram process is designed to organize a large number of notes captured
  during observations or interviews into a hierarchy to understand
  common issues and themes present in the data.
• The philosophy of affinity diagrams where issues and themes are derived
  from the data using a bottom-up approach was influenced by the grounded
  theory method, developed by Glaser and Strauss (1967).
• Grounded theory emphasizes building theories from the observed data rather than starting from preconceived hypothesis or theories. Researchers begin by conducting
  a microscopic examination of a subset of their data to identify concepts
  and categories in the data and relationships between them. These
  categories are then used and adjusted as needed while coding the rest of
  the data.