Please enable JavaScript.
Coggle requires JavaScript to display documents.
E-Lecture 8: Field Studies (Data gathering Logging Surveys …
E-Lecture 8: Field Studies
Learning objectives
-Distinguish between field studies and lab studies
Distinguish three common types of field study
Outline six data gathering techniques
Describe quantitative and qualitative analysis
Discuss how to make a study successful
What is a Field Study?
Field work (ethnography)
Lab studies (controlled)
Field studies
i.- You may already have met the concept of field work, probably from the perspective of ethnography
One of the characterising features of field work is that you immerse yourself in the natural environment of the participants
You may have seen how this can be adapted to the issue of design and the design process
The opposite of field work in some respect is a lab study
Lab studies are conducted in a controlled environment
Field study represents a kind of compromise between these two extreme approaches
The idea is to take some of the principles and approaches of lab studies into the natural environment of users
ii- Alternative Names
Field studies
In situ studies
In the wild studies
There are two other terms that you might hear for field studies: in situ studies and in the wild studies
In situ is looking for in the original place, meaning the study held in the original environment of the users
In the wild is used to contrast with in the lab. It means working with community, in their homes, in their workplaces and on the street
iii. Advantages of Field Studies
Ubiquitous technologies
Real world success
increasingly , studies are done in natural settings
This is likely respond to the development of ubiquitous technologies, which are developed for use outside office setting
The advantage of this approach is that you’re testing the technology in the real world
By evaluating how people think about interact and integrate product within the setting, they were also induce in getting a better sense of how successful the approach would be in the real environment
iv.Disadvantages of Field Studies
Little or no control
Difficult to test hypotheses
Interruption
There are 3 main disadvantages to conducting field studies
As an investigator, you have little or no control of the environment
It then become difficult or impossible to carry out hypothesis testing
The third disadvantage is the danger of interruption
When a mobile phone goes off, someone walks into the office or there is a loud noise in the adjoining room
Three types of field study
i. Current behaviour
ii. Proof of concept
iii. Experience of prototype
There are three common types of field studies: current behaviour studies, proof of concept studies and experience of using a prototype
i. Current behaviour (- What are people doing? -Shared technology example (Brush & Inkpen)
Current behaviour studies are concerned with what people are doing now
This type of field studies explores how people use existing technology
This kind of field studies help you understand current behaviour and the implication for future technology
Brush & Inkpen (2007)
Brush and Inkpen conducted a study in 2007 involving an interview with 15 family in the US
They examined the type of technology the families owned including television, music player, phones and computers
They also looked at where they were situated in their home and degree of ownership and use
During the visit, the participants were interviewed, they sketched the layout of the technologies in their home and gave the researchers tour of their home focusing on the location of the computers
There were some interesting observation about whether the users swap into their own profile to carry out certain task or just use someone else’s profile
They came up with some interesting recommendations about where the product should be designed from the outset for sharing
ii. Proof of Concept (Does it work in the real world?)
In a proof of concept study, the idea is to determine whether a new technology will function in the real world
he idea is to conduct a field study to validate the feasibility of an approach or prototype in the real world environment
Context Aware Power Management-Example
(Does it work in the real world?)
(Context-aware power management)
The next video is an example of a proof of concept study in the area of context-aware power management
Context-aware power management is about developing a technology to turn off your computer when you are away from it thus saving energy
This is not the study by Harrison Carhill that’s mentioned in the text, but I think is a very good example of a simplicity of a proof of concept study
iii. Experience using a prototype (Prototype & Wizard of Oz)
The third type of field studies is the experience of using a prototype
The idea is to get people using a prototype for long periods often weeks or months to understand how people used them
The use of the wizard of oz technique is particularly valuable here
Certain aspects of the prototype or systems are simulated where they would be otherwise be too expensive or time consuming to build
Design of Field studies
Hidden variables?
Within subjects
Qualitative comparison
In a conventional lab study, you control the environment so that the independent variable is the only thing that changes
In a field study, you have much less control of the environment , so you can’t be sure that the only thing affecting the dependent variable is the independent variable that you’re hoping to manipulate
There are likely to be other factors that might affect your dependent measure
For these reasons, a within the subject design is preferable over a between subject design
In a within the subject design, you involve the same people in the same circumstances
This reduces the number of possible hidden variables
The other advantage of within subject design, is it allows you to get the participants to create qualitative comparisons between the conditions
So for example, they could tell you which of two interfaces they preferred
Data gathering
Logging
Surveys
Experience sampling
Diaries
Interviews
Unstructured observation
We’re going to briefly consider six different techniques for gathering data
Some of them you’re already met before like the use of surveys or interviews
And others might be new to you such as logging or diaries
i. Logging (Main quantitative method, Prototype sends data)
Logging is probably the main method of collecting quantitative data in a field study
For example a prototype might write information to a data file when things occur that you want to know about
The next video gives an example of in-situ study of mobile phone notifications which uses logging as one of its data gathering techniques
ii. Survey
Presurvey
Postcondition
Postsurvey
3 main types of survey - Open-ended - Multiple-choice - Likert scale
There are 3 main types of survey used in a field study
A presurvey which is conducted before the study begins
A post condition survey which is used to examined the effects on each group of exposure to your system or prototype
And the postsurvey which is conducted at the end of the study
There are also a variety of different types of questions that might be used in a survey
I’m going to mention just 3
Open-ended questions where the users provide some written information or spoken information , to provide some qualitative data
Multiple choice questions where the users might have to choose among a number of options
Likert scales where the user has to express their agreement or level of agreement with the statement
Multiple choice questions and likert scale questions typically are used to provide quantitative data
Good and Bad Surveys
The next 3 videos are about good and bad surveys
The first is an example of a bad survey
When you watching this video, what I like you to do is to think about or maybe even jot down each of the mistake that were made in trying to conduct the survey
In the second video, Kenneth Fernandez from Elon University gives 7 tips for writing good surveys
The third video is about confirmation bias
Confirmation bias is about reaching a conclusion and then collecting data in order to confirm it
It often involves wording within a survey questions in such a way that the answer that you looking for is obvious
Survey suffering from confirmation bias avoid collecting data which might refute the conclusion that you’re already reached
iii. Diaries (Diary entries,User initiated)
The use of diaries is a similar technique to ESM
The users make an entry in the diary whenever they want to do something or feel certain way
Unlike ESM, entries in the diaries are initiated by the users rather than by the technology or investigator
iv. Interviews
-Semi-structured
-audio or video recording
-Photos
The use of semi-structured interview is also very common in field studies
Semi-structured interview involves taking an initial lists of questions into interview but following up with new questions if anyone has a conversation flows
Brushes recommendation is that you should make audio or video recording of interviews and take photos
Recording can be transcript or at least refer back to later
v. Unstructured observation
Little interaction with participants
Interviews are very useful field study technique
they may be less objective than observation because they involve interacting with the users which may affect what they say
Direct observation allows participants to be observed as they engaged in their normal lives
This might include interacting with a new technology or prototype without interference from the investigator
Data analysis
-Quantitative
-qualitative
In considering this technique, we have seen some of them have produced quantitative data and others produced qualitative data
Quantitative data involves number which are then analysed using statistical methods
Qualitative data involve words which have to be analysed using qualitative methods
i Statistics
-Descriptive statistics
-Inferential statistics
There are two kinds of statistics, descriptive statistics and inferential statistics
descriptive statistics are used to describe the sample which was involved in your studies
Inferential statistics involve looking at the sample as a way of representing a wider population of users
Inferential statistics involve using a statistical test to see whether your results can be generalised
ii. Qualitative Analysis
-Coding
-Pre-existing themes
-Testing theory (deduction)
-No pre-existing themes
-Deriving themes (induction)
-Theory building
Coding is the process of identifying common ideas or themes in the data
These might be pre-existing themes that come from theory which relates to the study
Looking through occurence of data relating to the theme, is known as deduction
Alternatively, the themes might emerge from the data
The process of generating the themes from the raw data is known as induction
The theme might then be used to build the theory
Experiments
-Study design
-Technology
-Running the study
-Data collection and analysis
Brush provides 4 different types of tips to make a successful study
Those relating to study design
Those relating to technology
Those relating to running the study
And those relating to data collection and analysis
i. Study Design
Clear research goal
Study design document
Scripts for visits
pilot
There are 4 tips relating to study design
The first is to have a clear research goal, by this we mean having a very clear research question
Without having a a very clear research question, it might be hard to explain your finding and why the make contribution to knowledge
Brush also makes the point that if you’re not clear about what’s your question is, there is danger that you end up analysing the usability problem of the prototype
But the usability problems of your prototypes are of no interest to other researchers
The second tip is to have a study design document, which is like a plan for your methodology
It should include things like your research questions, who your participants is going to be, your methodology is going to be, and how many times you going to make a visit and collect data, and how you might analyse the data
It’s the kind of thing that you ultimately put into the methodology sections of an academic report
The third tip is to make a script for the participant visit, this is just a plan of what is going to happen when you arrive
It can serve as a reminder to make sure you do all the things that you need to do, raw checklist to make sure that they all being done
The final tip is to conduct a pilot study
A pilot study is like a dress rehearsal of the real thing
You have normally do it with your colleagues or friends so that they can give you some useful feedback before you go into the field
ii. Technology
Robust enough technology
Other evaluation methods
Existing technology
Reassuring feedback
Negative results
Brush gives 5 tips relating to technology
The first is to ensure that your technology is robust enough that means it good enough to provide you with the data to answer your research questions
But they may be things about it that don’t work
The second tip is to consider using other evaluation methods before you go into the field, Brush gives the example of heuristics evaluation or in laboratory study
The next tip is to use existing technology to make your life easier. There are various commercially available prototyping kit that might help you
The fourth tip is to get feedback during your study that reassure you that everything is working, don’t wait until the end of interview for example to discover that your microphone is not working
The final tip is to have a plan B, the plan B is what you would do if things don’t quite go according to plan, in other words, how to deal with negative results
iii. Running the study
Research team
Comfortable participants
Safety
Flexibility
The next four tips relate to running the study
The first tip is to put together a research team to make the study more manageable
The second tip is to make sure your participants are comfortable, use some small talk to put them at their ease
The next tip is to make sure that you take the safety of both the participants and you as the investigator into consideration
Brush suggests gender mixed team when visiting people in their home particular women who may be alone
The final tip is to be flexible. In field studies, things often go wrong, so be prepared to re-organise things if necessary
iv. Data collection and analysis
Be objective
The participant is always right
Inappropriate claims
Finally, there are three tips relating to data collection and analysis
The first is to be objective, be careful to avoid the situation where participants tell you what they think you want to hear
The second tip is not to argue with participants
Don’t get defensive if they criticise something you’ve spent week developing or if they describe something as a problem when you know that it isn’t
The participant is always right, make a note of it, and think about it later
The third and final tip is to recognise that your study would have its limitations and you shouldn't make inappropriate claims
For example, rather than saying the results prove that such and such, use words like suggest or support