19 The evaluation of accessibility, usability, and user experience

abstract

eSystem should be easy and pleasant to use for the target audience

target audience

broadest

disabilities & older people

specific

students in biology

outline

concepts of accessibility, usability and user experience

the iterative user-centred design lifecycle

a range of methods for evaluating accessibility & usability & user experience

strengths & weaknesses & appropriate use

  1. Accessibility, usability and user experience

term

user friendly

easy to use

usability

ISO 9241

The extent to which a product [service or environment] can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

✅ Effectiveness

✅ efficiency

✅ satisfaction

many practitioners

Gould and Lewis 1985; Shackel, 1990; 1991; Sharp, Rogers and Preece 2007; Stone et al. 2005

✅ flexibility

✅ learnability

✅ memorability

✅ safety

show

✅ users

✅ goals

✅ contexts

accessibility

ISO 9241-171

the usability of a product, service, environment or facility by people with the widest range of capabilities

✅ usability is a sub-set of accessibility

WAI

people with disabilities can use the Web. More specifically, Web accessibility means that people with disabilities can perceive, understand, navigate, and interact with the Web

✅ accessibility as a sub-set of usability

now

usability -> mainstream (i.e. non-disabled, younger) users

accessibility -> disabled and older users

universal design

design for all

User experience

Norman

Hassenzah

Dillon (2001)

user experience, initially popularized by him in 1998

users’ interactions with, and reactions to, eSystems that go beyond effectiveness, efficiency, and conventional interpretations of satisfaction.

✅ Holistic

✅ Subjective

✅ Positive

✅ Process

✅ Outcomes

✅ affect

Bevan (2008)

✅ Likability

✅ Pleasure

✅ Comfort

✅ Trust

outline

important part of evaluating eSystem & more important in the future

task-oriented aspects

other non-task oriented aspects (hedonic aspects)

beauty

challenge

stimulation

self-expression

usability (objective)

percentage of tasks achieved for effectiveness

task completion times and error rates for efficiency

UX (sujective)

users’ subjective reactions to eSystem

their perceptions of the eSystems

their interaction with them

usability (negative)

removal of barriers or problems

UX (positive)

maximize positive aspects of eSystem

an understanding of users’ moves, attention and difficulties through an eSystem

an understanding of what it means for the user to feel accomplishment or closure with the eSystem

an understanding of users’ emotional interaction with eSystems and what interaction means for users

new ISO Draft International Standard 9241-210

A person's perceptions and responses that result from the use or anticipated use of a product, system or service

  1. Design and evaluation processes
    iterative user-centred design and inclusive design

3.1 Iterative, user-centred design

9-1

✅ Understanding users, tasks, contexts

✅ Design

✅ Prototype

✅ Integration & final implementation

methods

✅ study existing guidelines

✅ interview current & potential users

✅ conduct an ethnographic & context of use

record in a user requirements document

track subsequent design & development

update to reflect changes in understanding user requirements

Common Industry Specification for Usability Requirements

✅ explore the design space

❌ immediately settling on one design

forms

simple & complex

sketches on paper & animations with interactivity

purpose

people involve in evaluation

address functionality of eSystem

explore particular design problems

whether feasible or not

feed back into design process

✅ Evaluation

⭐ heart of the process

can be done by

experts

users

purpose

track

feed back

people are such complex entities

process

20+ iterations

main design & development -> a month -> 3~4 iteration

preparation -> long time -> 15+ iterations

go on in parallel with back-end development

usability, accessibility and user experience

✅ evaluation in real context

feed back

understanding users

design process

this version & subsequent versions

3.2 Inclusive design

process

❌ consider accessibility in the end -> impossible & cost max

term

✅ consider accessibility from beginning -> cost min

✅ universal design

✅ design for all

✅ barrier free design

✅ inclusive design

the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design.

ideal & practice

❌ can't be met in practice

❌ frighten designers and developers

✅ not need to solve all the accessibility problems

assistive technologies

✅ screen readers

✅ screen magnification programs

✅ a variety of alternative input devices

✅ ...

user group

✅ don't use an assistive technology

✅ do use an assistive technology

✅ can not use the system without an assistive technology

integrated into the iterative user-centred design process

  1. Methods for evaluation

outline

✅ Automated checking of conformance to guidelines and standards

✅ Evaluations conducted by experts

✅ Evaluations using models and simulations

✅ Evaluation with users or potential users

✅ Evaluation of data collected during eSystem usage

4.1 Guidelines for accessibility and usability

4.1.1 Accessibility Guidelines

IBM

Software •
Websites and applications
Java applications
Lotus notes
Hardware
Documentation

3 ISO standards

✅ ISO/IEC 610779 (2008d) -> Office equipment accessibility guidelines for elderly persons and persons with disabilities

✅ ISO 9241-20 (2008a) -> Ergonomics of human-system interaction – Part 20: Accessibility guidelines for information/communication technology (ICT) equipment and services

✅ ISO 9241-171 (2008b) -> Ergonomics of human-system interaction – Part 171: Guidance on software accessibility

WAI

WCAG 1.0 1999

P1 & P2 & P3

14 high level accessibility guidelines

65 specific checkpoints

WCAG 2.0 2008

A & AA & AAA

✅ perceivable

✅ operable

✅ understandable

✅ robust

Section 508

Section 508 of the Rehabilitation Act of the United States Federal government

Human Ability and Accessibility Centre

technologies

✅ Adobe Flash™

Regan (2004) and resources at the Adobe Accessibility Resource Centre

✅ Content Management Systems Joomla!

O’Connor (2007)

✅ Eclipse open source development platform

Accessibility Features in Eclipse 9and the Eclipse Accessibility Tools Framework Project

✅ Java

resources at Sun Accessibility 11

✅ Microsoft™ products

resources at the Microsoft Accessibility Developer Center

4.1.2 Usability Guidelines and Standards

Nielsen

✅ Visibility of system status

✅ Match between system and the real world

✅ User control and freedom

✅ Consistency and standards

✅ Error prevention

✅ Recognition rather than recall

✅ Flexibility and efficiency of use

✅ Aesthetic and minimalist design

✅ Help users recognize, diagnose, and recover from errors

✅ Help and documentation

Shneiderman

✅ Strive for consistency

✅ Enable frequent users to use shortcuts

✅ Offer informative feedback

✅ Design dialogue to yield closure

✅ Offer simple error handling

✅ Permit easy reversal of actions

✅ Support internal locus of control

✅ Reduce short-term memory load

U.S. Government Department of Health and Human Services (HHS)

207 guidelines

ISO 9241 standard

very time consuming to employ in evaluation

need to be interpreted in relation to a particular interface environment

more complete and easier to use than the equivalent ISO

4.1.3 Problems with guidelines

problems

✅ learn -> large number of guidelines & efforts

✅ evaluation -> time consuming

✅ general guidelines -> not apply particular circumstances

✅ difficult to apply without experience in application domain

user testing

✅ user type

✅ task

❎ environmental conditions

rating & accessibility issue

no correlation in the ratings given by both disabled and mainstream users of actual problems

4.2 Automated checking of conformance to guidelines or standards

prototypes & full implementations

purpose

not contain basic accessibility and usability problems

tools for automated accessibility checking

❌ Bobby Tool -> no longer available

❌ Many WCAG Checkpoints cannot be checked automatically -> 23% can

❌ no automatic tool can check whether the alternative descriptions are accurate and useful
blabla -> forgotten placeholder
elephant -> cat mistake placeholder
person at computer -> why include the image

tools for automated usability checking

✅ LIFT

✅ Web Static Analyzer Tool

❌ basic problems & limited scope of usability issues

4.3 Evaluations conducted by experts

prototypes

purpose

identify more accessibility issues before user-based evaluations

too many pages / screens to user-based evaluation

not possible to obtain actual users

insufficient time for user testing

train developers

type

✅ non task scenario

review

inspection

✅ task scenario

walkthrough

9-2

⭐ Heuristic evaluation

a small set of evaluators -> potential usability problems -> Cello evaluation

✅ evaluators
✅ a set of heuristics
✅ rate the potential problems individually -> how severe
✅ a four level rating scheme is used (1,2,3,4)
✅ better results are obtained by trained experts

✅ asking the evaluator to step through typical user tasks
heuristic evaluation + cognitive walkthrough

✅ heuristics are a useful training aid & value is not clear

⭐ Expert Walkthrough Evaluations

identifies problems while attempting to achieve tasks in the same way as a user

✅ Cognitive walkthrough

each user action -> evaluator analyses
interface
feedback

✅ Pluralistic walkthrough

user & developer & human factor people

a scenario & discuss & dialogue

⭐ Expert evaluations for accessibility

✅ websites and web-based applications

Heuristic evaluation

Expert Walkthrough Evaluations

✅ other types of eSystems

on an ad hoc basis by an expert

employ user evaluation instead

preliminary accessibility review

✅ a representative set of pages

✅ a range of graphical browsers

  1. turn off images
  2. turn off the sound
  3. use browser controls to vary font-size
  4. test with different screen resolution
  5. change the display colour to greyscale
  6. without using the mouse

tool

✅ IE & Opera

AIS Toolbar

✅ Firefox

Accessibar add-on

WAVE Toolbar

✅ Screenreader

JAWS

WindowEyes

period

begin

preliminary accessibility review can guide the further development

complete

a full accessibility audit

⭐ Unified Web Evaluation Methodology

✅ a standard reporting format for accessibility evaluation reports

✅ UWEM and WAI websites

4.4 Evaluations using models and simulations

✅ simpler and quicker

✅ a wider range of users and tasks

❌ emphasize more superficial problems

❌ not scale well for complex interfaces

✅ difference between experts & users -> reliable of results

when

models can be constructed economically

user testing is not practical

purpose

time to complete tasks is critical

type

✅ Keystroke Level Model

✅ Goals Operators Methods and Selections (GOMS) model

✅ ACT-R model

other methods are impracticable

information provided by the model is a effective way to managing risks

4.5 Evaluations with users

when

✅ max -> all stages of development,

✅ min -> the final stage of development

purpose

users

developers & managers

sample size

✅ no way to find all usability issues in one evaluation

sample size

✅ previous similar usability evaluation

✅ initial results of an ongoing study

probabilities

✅ 0.08~0.42

✅ 3~19 -> 80% problems

✅ 4~28 -> 90% problems

✅ complex website -> lower probabilities

process

✅ Iterative testing with small numbers of participants is preferable

✅ test with at least 8 users

type

9-3

✅ formative

✅ summative

Formative methods focus on understanding the user’s behaviour, intentions and expectations in order to understand any problems encountered, and typically employ a “think-aloud” protocol;

Summative methods measure the product usability or accessibility, and can be used to establish and test user requirements

ethnics

ways

✅remote evaluation

✅ simulate users with disabilities -> disabled develop strategies to deal with problems -> not comparable

consider

✅ assistive technology

✅ expensive & time consuming -> remote testing instead

✅ the needs of participants in the evaluation

✅ how will they come

✅ the location accessible

✅ explanatory material & consent forms -> alternative formats

✅ pace of evaluation suitable

UX & satisfaction

✅ Kansei techniques

✅ rating scales and questionnaires -> simplest

✅ Psychometrically designed questionnaires -> SUS -> usability

✅ AttrakDiff -> user experience

4.6 Evaluation of data collected during eSystem usage

when

improve an existing eSystem

how

Provides non-intrusive data

✅ Satisfaction Surveys

✅ Web server log analysis

✅ Application Instrumentation

entrance and exit pages

economical & feedback

frequency of particular paths through the site

the extent to which search it successful

difficult to track individual behaviour

particular user groups and tasks

data point

anonymously to developer

guide future design decision

  1. Conclusion

start with some of the simpler expert evaluations and simple user-based evaluations

consensus
abbreviated
portable
conventional
contradictory
holistic
subjective
anticipated
encompass
hedonic aspects
attain
constitutes
closure
empowered
situate
phase
spiral
explicitly
ethnographic
encapsulate
specification
contradicting
retro-fitting
coin
literal
lotus
legislation
federal
heuristics
Accelerators
diminishes
Strive
yield
reversal
locus
substantial
generalization
eventual
permutation
eliminate
walkthrough
review
inspection
Pluralistic
Preliminary
dispense
lightweight
sophisticated
audit
cost-effective
formative
summative
probabilities
iterative test
Diagnostic
fidelity
benchmark
validate
personnel
explanatory
psychometrically
intrusive
economical








共识
简短的
手提
常规
矛盾
整体
主观
预期
环绕
享乐方面
达到
构成
关闭
授权
位于

螺旋
明确地
人种学
封装
规范
矛盾
加装
硬币
文字
莲花
立法
联邦
启发式
加速器
减少
努力
产量
翻转
轨迹
大量的
概括
最终
排列
消除
演练
评论
检查
多元
初步
发放
轻量级
复杂的
审计
经济有效
形成性
总结
概率
迭代测试
诊断
保真度
基准
验证
人员
解释性
心理测量
侵入
经济