Please enable JavaScript.

Coggle requires JavaScript to display documents.

Evaluation, Types of Evaluations - Coggle Diagram

- - - - rather have multiple test runs with the same participants than only one test run with lots of participants
      - rather 3 users , 2 runs , than 9 users 1 run
  - - - help participants get into subject more easily
    - - tasks should answer questions you have regarding interface design
      - start with something simple
      - formulated in user language (not system)
      - tasks should not implay specific set of steps to be taken
  - - - performance data
        
        measurements of users action
        
        time
        
        number of errors
        
        recovery from errors
        
        succes rate / fail rate
      - preference daat
        
        users responses on post task questionnaires
- - - - applied experiment
      - not completely replicable
      - conditions controlled as much as possible
      - goal: improve product
      - fewer participants
      - results inform the deesign
      - results reported to developers
      - Measurements
        
        Performance based
        
        Task-success
        
        Time on Task
        
        Errors
        
        Efficiency/Effectiveness
        
        Effectiveness = #succ tasks / #total tasks * 100%
        
        Efficiency t = start t - end t
        
        time based efficiency
        
        overall relative Efficiency
        
        Learnability
        
        Issue-based metrics
        
        Frequency of unique issues
        
        frequency of issues per participant :
        
        Issues by category or task
        
        self.reported metrics
        
        rating scales
        
        NASA
        
        subj workload assessment
        
        SEQ
        
        Single Ease Question
        
        7-point rating scale
        
        how difficult users find task
        
        Test Level Satisfaction
        
        SUS
        
        SUPR-Q
        
        QUIS
        
        SUMI
        
        UMEX
        
        UEQ
        
        AttrakDiff2
        
        after scenario Questionnaires
        
        Assessing specific attributes
        
        Open Questions
        
        behavioural and physiological metrics
        
        Eye tracking
        
        mesuring stress
    - - psychology-style
      - compare different designs
      - goal: discover knowlegde
      - must be replicable
      - Settings are strongly controlled
      - scientific report to scientific community
  - - - eher wenige Probanden
      - more qualitative data
      - Warum?
    - - self-moderated
        
        User reported Critical Incidents (UCI)
        
        relieves evaluators from conducting tests and anylzing results
        
        quality of problem description depends on given form
        
        unstructured problem reporting
        
        easy to implement
        
        usability feedback might be very unstructured and quite difficult to evaluate
        
        form-based online reporting and forum
        
        diary based lonigtudinal reporting
        
        users work with system bc tasks are regularly distributed
        
        analysis of diary entries require a lot of time
        
        Nutzung d Produkts über längeren Zeitraum
      - automated
        
        quantitaive data
        
        e.g. Auto Logging
        
        easy with ready to go tools !
        
        can't collect qualitative data
        
        necessary to combine with interviews or other method to get qualitative data
        
        Metriken
        
        viele Teilnehmer
      - Tester and evaluators are seperated in space and time
- - - - formalized way of imagining people's thoughts and actions
      - more sequence like view
      - Process
        
        1 Beschreibung d Prototypischen Interfaces (detailliert!)
        
        2 Task beschreibungen
        
        3 Complete written List of actions needed to complete task (als Ablaufdiagramm)
        
        4 Idea of who users will be & what they bring
      - verstehen wie gut ein System erlenbar ist
      - sichtbar wie viele falsche actions passieren
      - Mühsam und sehr hoher detailgrad erforderlich
    - - Nielsens Heuristic
        
        1 Visibility of System Status
        
        Feedback
        
        wie gut vermittelt System seinen status an User
        
        2 Match bewteen User and real World
        
        Metaphor
        
        Spricht das SWSys Sprache d Nutzerin
        
        3 User control and freedom : :
        
        navigation
        
        Notausgang zB "undo"
        
        4 consistency and standards
        
        consistency
        
        users should not have to wonder if different words mean the same functionality
        
        6 recognition rather than recall
        
        memory
        
        minimize users memory load
        
        5 error prevention
        
        prevention
        
        even better than error message, prevent those
        
        pre conditions, confirmation
        
        7 flexibility and efficiency to use
        
        efficiency
        
        accelarators unseen by normal user e.g. shortcuts
        
        8 aesthetic and minimalistic design
        
        design
        
        no irrelevant information, diminishes relevant information
        
        9 help users recognize, diagnose, recover from errors
        
        recovery
        
        msg should be in plain language and helpful to solve problem
        
        10 help and documentation
        
        provide help and documentation
      - systematic inspection of user interface design for usability
        
        goal: find usability problems
      - Pros & Cons
        
        Pros
        
        no user involvement
        
        identifies major prblm well
        
        easy to learn to do
        
        can train devs to do it
        
        cheap
        
        gives pointers for improvements
        
        can be applied to interfaces in varying states of readyness
        
        Cons
        
        relies on heuristics
        
        only as good as evaluator applying it
        
        broad-based guidelines
        
        does not involve users
        
        can't cover every type of interface that might get built
      - Process
        
        prepare
        
        evaluate alone
        
        discuss findings in group
        
        document
      - Severity
        
        Frequency of appearance
        
        impact on users
        
        persistence: does it need to be overcome regularly
  - - - set of GOALS
      - set of OPERATORS
      - set of METHODS for achieving goals
      - set of SELECTION rules
      - ..to choose among competing methods for a goal
      - How much time does a skilled user needs?
    - - Key stroke level Model
      - quantitaive analysis tool for predicting user performance
      - based on timing of different gestures, which has been developed in studies