19 The evaluation of accessibility, usability, and user experience
abstract
eSystem should be easy and pleasant to use for the target audience
target audience
broadest
disabilities & older people
specific
students in biology
outline
concepts of accessibility, usability and user experience
the iterative user-centred design lifecycle
a range of methods for evaluating accessibility & usability & user experience
strengths & weaknesses & appropriate use
- Accessibility, usability and user experience
term
user friendly
easy to use
usability
ISO 9241
The extent to which a product [service or environment] can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.
✅ Effectiveness
✅ efficiency
✅ satisfaction
many practitioners
Gould and Lewis 1985; Shackel, 1990; 1991; Sharp, Rogers and Preece 2007; Stone et al. 2005
✅ flexibility
✅ learnability
✅ memorability
✅ safety
show
✅ users
✅ goals
✅ contexts
accessibility
ISO 9241-171
the usability of a product, service, environment or facility by people with the widest range of capabilities
✅ usability is a sub-set of accessibility
WAI
people with disabilities can use the Web. More specifically, Web accessibility means that people with disabilities can perceive, understand, navigate, and interact with the Web
✅ accessibility as a sub-set of usability
now
usability -> mainstream (i.e. non-disabled, younger) users
accessibility -> disabled and older users
universal design
design for all
User experience
Norman
Hassenzah
Dillon (2001)
user experience, initially popularized by him in 1998
users’ interactions with, and reactions to, eSystems that go beyond effectiveness, efficiency, and conventional interpretations of satisfaction.
✅ Holistic
✅ Subjective
✅ Positive
✅ Process
✅ Outcomes
✅ affect
Bevan (2008)
✅ Likability
✅ Pleasure
✅ Comfort
✅ Trust
outline
important part of evaluating eSystem & more important in the future
task-oriented aspects
other non-task oriented aspects (hedonic aspects)
beauty
challenge
stimulation
self-expression
usability (objective)
percentage of tasks achieved for effectiveness
task completion times and error rates for efficiency
UX (sujective)
users’ subjective reactions to eSystem
their perceptions of the eSystems
their interaction with them
usability (negative)
removal of barriers or problems
UX (positive)
maximize positive aspects of eSystem
an understanding of users’ moves, attention and difficulties through an eSystem
an understanding of what it means for the user to feel accomplishment or closure with the eSystem
an understanding of users’ emotional interaction with eSystems and what interaction means for users
new ISO Draft International Standard 9241-210
A person's perceptions and responses that result from the use or anticipated use of a product, system or service
- Design and evaluation processes
iterative user-centred design and inclusive design
3.1 Iterative, user-centred design
✅ Understanding users, tasks, contexts
✅ Design
✅ Prototype
✅ Integration & final implementation
methods
✅ study existing guidelines
✅ interview current & potential users
✅ conduct an ethnographic & context of use
record in a user requirements document
track subsequent design & development
update to reflect changes in understanding user requirements
Common Industry Specification for Usability Requirements
✅ explore the design space
❌ immediately settling on one design
forms
simple & complex
sketches on paper & animations with interactivity
purpose
people involve in evaluation
address functionality of eSystem
explore particular design problems
whether feasible or not
feed back into design process
✅ Evaluation
⭐ heart of the process
can be done by
experts
users
purpose
track
feed back
people are such complex entities
process
20+ iterations
main design & development -> a month -> 3~4 iteration
preparation -> long time -> 15+ iterations
go on in parallel with back-end development
usability, accessibility and user experience
✅ evaluation in real context
feed back
understanding users
design process
this version & subsequent versions
3.2 Inclusive design
process
❌ consider accessibility in the end -> impossible & cost max
term
✅ consider accessibility from beginning -> cost min
✅ universal design
✅ design for all
✅ barrier free design
✅ inclusive design
the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design.
ideal & practice
❌ can't be met in practice
❌ frighten designers and developers
✅ not need to solve all the accessibility problems
assistive technologies
✅ screen readers
✅ screen magnification programs
✅ a variety of alternative input devices
✅ ...
user group
✅ don't use an assistive technology
✅ do use an assistive technology
✅ can not use the system without an assistive technology
integrated into the iterative user-centred design process
- Methods for evaluation
outline
✅ Automated checking of conformance to guidelines and standards
✅ Evaluations conducted by experts
✅ Evaluations using models and simulations
✅ Evaluation with users or potential users
✅ Evaluation of data collected during eSystem usage
4.1 Guidelines for accessibility and usability
4.1.1 Accessibility Guidelines
IBM
Software •
Websites and applications
Java applications
Lotus notes
Hardware
Documentation
3 ISO standards
✅ ISO/IEC 610779 (2008d) -> Office equipment accessibility guidelines for elderly persons and persons with disabilities
✅ ISO 9241-20 (2008a) -> Ergonomics of human-system interaction – Part 20: Accessibility guidelines for information/communication technology (ICT) equipment and services
✅ ISO 9241-171 (2008b) -> Ergonomics of human-system interaction – Part 171: Guidance on software accessibility
WAI
WCAG 1.0 1999
P1 & P2 & P3
14 high level accessibility guidelines
65 specific checkpoints
WCAG 2.0 2008
A & AA & AAA
✅ perceivable
✅ operable
✅ understandable
✅ robust
Section 508
Section 508 of the Rehabilitation Act of the United States Federal government
Human Ability and Accessibility Centre
technologies
✅ Adobe Flash™
Regan (2004) and resources at the Adobe Accessibility Resource Centre
✅ Content Management Systems Joomla!
O’Connor (2007)
✅ Eclipse open source development platform
Accessibility Features in Eclipse 9and the Eclipse Accessibility Tools Framework Project
✅ Java
resources at Sun Accessibility 11
✅ Microsoft™ products
resources at the Microsoft Accessibility Developer Center
4.1.2 Usability Guidelines and Standards
Nielsen
✅ Visibility of system status
✅ Match between system and the real world
✅ User control and freedom
✅ Consistency and standards
✅ Error prevention
✅ Recognition rather than recall
✅ Flexibility and efficiency of use
✅ Aesthetic and minimalist design
✅ Help users recognize, diagnose, and recover from errors
✅ Help and documentation
Shneiderman
✅ Strive for consistency
✅ Enable frequent users to use shortcuts
✅ Offer informative feedback
✅ Design dialogue to yield closure
✅ Offer simple error handling
✅ Permit easy reversal of actions
✅ Support internal locus of control
✅ Reduce short-term memory load
U.S. Government Department of Health and Human Services (HHS)
207 guidelines
ISO 9241 standard
very time consuming to employ in evaluation
need to be interpreted in relation to a particular interface environment
more complete and easier to use than the equivalent ISO
4.1.3 Problems with guidelines
problems
✅ learn -> large number of guidelines & efforts
✅ evaluation -> time consuming
✅ general guidelines -> not apply particular circumstances
✅ difficult to apply without experience in application domain
user testing
✅ user type
✅ task
❎ environmental conditions
rating & accessibility issue
no correlation in the ratings given by both disabled and mainstream users of actual problems
4.2 Automated checking of conformance to guidelines or standards
prototypes & full implementations
purpose
not contain basic accessibility and usability problems
tools for automated accessibility checking
❌ Bobby Tool -> no longer available
❌ Many WCAG Checkpoints cannot be checked automatically -> 23% can
❌ no automatic tool can check whether the alternative descriptions are accurate and useful
blabla -> forgotten placeholder
elephant -> cat mistake placeholder
person at computer -> why include the image
tools for automated usability checking
✅ LIFT
✅ Web Static Analyzer Tool
❌ basic problems & limited scope of usability issues
4.3 Evaluations conducted by experts
prototypes
purpose
identify more accessibility issues before user-based evaluations
too many pages / screens to user-based evaluation
not possible to obtain actual users
insufficient time for user testing
train developers
type
✅ non task scenario
review
inspection
✅ task scenario
walkthrough
⭐ Heuristic evaluation
a small set of evaluators -> potential usability problems -> Cello evaluation
✅ evaluators
✅ a set of heuristics
✅ rate the potential problems individually -> how severe
✅ a four level rating scheme is used (1,2,3,4)
✅ better results are obtained by trained experts
✅ asking the evaluator to step through typical user tasks
heuristic evaluation + cognitive walkthrough
✅ heuristics are a useful training aid & value is not clear
⭐ Expert Walkthrough Evaluations
identifies problems while attempting to achieve tasks in the same way as a user
✅ Cognitive walkthrough
each user action -> evaluator analyses
interface
feedback
✅ Pluralistic walkthrough
user & developer & human factor people
a scenario & discuss & dialogue
⭐ Expert evaluations for accessibility
✅ websites and web-based applications
Heuristic evaluation
Expert Walkthrough Evaluations
✅ other types of eSystems
on an ad hoc basis by an expert
employ user evaluation instead
preliminary accessibility review
✅ a representative set of pages
✅ a range of graphical browsers
- turn off images
- turn off the sound
- use browser controls to vary font-size
- test with different screen resolution
- change the display colour to greyscale
- without using the mouse
tool
✅ IE & Opera
AIS Toolbar
✅ Firefox
Accessibar add-on
WAVE Toolbar
✅ Screenreader
JAWS
WindowEyes
period
begin
preliminary accessibility review can guide the further development
complete
a full accessibility audit
⭐ Unified Web Evaluation Methodology
✅ a standard reporting format for accessibility evaluation reports
✅ UWEM and WAI websites
4.4 Evaluations using models and simulations
✅ simpler and quicker
✅ a wider range of users and tasks
❌ emphasize more superficial problems
❌ not scale well for complex interfaces
✅ difference between experts & users -> reliable of results
when
models can be constructed economically
user testing is not practical
purpose
time to complete tasks is critical
type
✅ Keystroke Level Model
✅ Goals Operators Methods and Selections (GOMS) model
✅ ACT-R model
other methods are impracticable
information provided by the model is a effective way to managing risks
4.5 Evaluations with users
when
✅ max -> all stages of development,
✅ min -> the final stage of development
purpose
users
developers & managers
sample size
✅ no way to find all usability issues in one evaluation
sample size
✅ previous similar usability evaluation
✅ initial results of an ongoing study
probabilities
✅ 0.08~0.42
✅ 3~19 -> 80% problems
✅ 4~28 -> 90% problems
✅ complex website -> lower probabilities
process
✅ Iterative testing with small numbers of participants is preferable
✅ test with at least 8 users
type
✅ formative
✅ summative
Formative methods focus on understanding the user’s behaviour, intentions and expectations in order to understand any problems encountered, and typically employ a “think-aloud” protocol;
Summative methods measure the product usability or accessibility, and can be used to establish and test user requirements
ethnics
ways
✅remote evaluation
✅ simulate users with disabilities -> disabled develop strategies to deal with problems -> not comparable
consider
✅ assistive technology
✅ expensive & time consuming -> remote testing instead
✅ the needs of participants in the evaluation
✅ how will they come
✅ the location accessible
✅ explanatory material & consent forms -> alternative formats
✅ pace of evaluation suitable
UX & satisfaction
✅ Kansei techniques
✅ rating scales and questionnaires -> simplest
✅ Psychometrically designed questionnaires -> SUS -> usability
✅ AttrakDiff -> user experience
4.6 Evaluation of data collected during eSystem usage
when
improve an existing eSystem
how
Provides non-intrusive data
✅ Satisfaction Surveys
✅ Web server log analysis
✅ Application Instrumentation
entrance and exit pages
economical & feedback
frequency of particular paths through the site
the extent to which search it successful
difficult to track individual behaviour
particular user groups and tasks
data point
anonymously to developer
guide future design decision
- Conclusion
start with some of the simpler expert evaluations and simple user-based evaluations
consensus
abbreviated
portable
conventional
contradictory
holistic
subjective
anticipated
encompass
hedonic aspects
attain
constitutes
closure
empowered
situate
phase
spiral
explicitly
ethnographic
encapsulate
specification
contradicting
retro-fitting
coin
literal
lotus
legislation
federal
heuristics
Accelerators
diminishes
Strive
yield
reversal
locus
substantial
generalization
eventual
permutation
eliminate
walkthrough
review
inspection
Pluralistic
Preliminary
dispense
lightweight
sophisticated
audit
cost-effective
formative
summative
probabilities
iterative test
Diagnostic
fidelity
benchmark
validate
personnel
explanatory
psychometrically
intrusive
economical
共识
简短的
手提
常规
矛盾
整体
主观
预期
环绕
享乐方面
达到
构成
关闭
授权
位于
相
螺旋
明确地
人种学
封装
规范
矛盾
加装
硬币
文字
莲花
立法
联邦
启发式
加速器
减少
努力
产量
翻转
轨迹
大量的
概括
最终
排列
消除
演练
评论
检查
多元
初步
发放
轻量级
复杂的
审计
经济有效
形成性
总结
概率
迭代测试
诊断
保真度
基准
验证
人员
解释性
心理测量
侵入
经济