Please enable JavaScript.
Coggle requires JavaScript to display documents.
Week 6.1 - Rigour, Reliability and Validity (Qualitative (for rigour we…
Week 6.1 - Rigour, Reliability and Validity
Rigour:
- being extremely thorough & accurate
- enhances believability, trustworthiness and quality of the research
- required to ensure the study findings are transferable (qualitative) and generalizable (quantitative)
- tries to get rid of confounding variables, deal with bias, use different designs to get quality results
- TERMS: (similar terms for the different paradigms or methodology)
- Qualitative (credibility = true value; transferability = applicability, "fittingness"; dependability = consistency; confirmability = neutrality, audit-ability)
- Quantitative (internal validity; external validity/generalizability; reliability; objectivity)
Qualitative (for rigour we look at trustworthiness):
- Trustworthiness:
- credibility: truth of findings as judged by participants and others within the discipline
- transferability/fittingness: (do not say results are generalizable); extent to which findings can be generalized to other situations; faithfulness to everyday reality of the participants - "rings true"; thick description (rich description so findings can be transferable; quotes, what people are saying)
- dependability: process of study inquiry is above board
- audit-ability (confirmability): adequacy of information leading the reader from research question and raw data to the interpretation of findings (good manuscript will tell reader how it was done and get them to land on the same conclusions)
Credibility: How accurately have you recorded the findings?
- most important one
- to increase probability of high credibility (prolonged engagement, prolonged observation, triangulation of sources, methods, investigators); peer debriefing (talk to experts in the field about the results)
- negative case analysis (look at outliers and try to understand them)
- member checks (reconfirm with what participants are telling you)
- always look at credentials; minimize bias, researcher reflectivity
Transferability:
- extent to which findings can be applied to other situations (but not generalizable)
- need to make context very clear to increase transferability
- Detail every step of the study !!!
- readers interpretation and application to their own life - does it have relevance for them?
- compare and contrast
Dependability:
- examination of process by which study was done (external auditor)
- ensuring results come from study participants and not from characteristics or preferences of the researcher
acknowledges limitations so people are aware
- diagrams of audit trails (keep good results and stay organized)
Confirmability (auditability)
- audit trails (details of research design, implementation, operational details; implementation of study etc)
- are the theories that are developed coming from the data or from researchers reflections on data?
- reflective appraisals (memos, field notes, researcher journal) --> want other researchers to be able to follow your way of thinking and a way to get rid of bias
Reliability: (Quantitative world --> rigour means reliability)
- consistency and accuracy with which an instrument measures the target attribute
- reliability assessments involve computing a reliability coefficient (most reliability coefficients are based on correlation coefficients - is there an association here)
Reliability Coefficients:
- indicate magnitude of relationships between variables
- range (0 = very high error, no reliability) // (1 = no error, high reliability)
- for a tool to be considered reliable, a level of 0.70 or higher should be reported
- a good article will tell you what reliability and validity is
Measurement Tool:
- whole thing together (ex. survey or questionnaire)
- you have scales within instrument (may have 1 or multiple scales)
- scales are groups of items asking about the same thing (most range rom 6-8 items)
Three Aspects of Reliability Can Be Evaluated:
- Stability
- Internal Consistency
- Equivalence
Stability:
- extent to which scores are similar on 2 separate administrations of an instrument
- evaluated by:
- test-retest reliability (focus on this one as more common; giving the test twice)
- parallel form reliability (requires participants to complete same instrument or parallel form on 2 occasions; correlation coefficient computed between the 2 scores; appropriate for relatively enduing attributes - ex. self-esteem)
Internal Consistency (Homogeneity):
- extent to which all the instrument's items are measuring the same attribute (are we sure they are testing the same and proper thing that we are wanting to test?)
- evaluated by administering instrument on one occasion; appropriate for multi-item instruments
- Methods:
- Item-to-Total Correlation: statistical test to create a matrix to see how correlated each item is to the total (want correlated between 0.3-0.8; balancing act to get them correlated but each item is getting at a slightly different aspect; want different approaches to your construct)
- Split-Half Reliability: splitting all the items in the scale in half and comparing the halves – are the scores for each half no matter how you split them about the same? (no matter how you split them)
- Kuder-Richarson Coefficient: used when questions have a dichotomous response (dichotomous is one or the other); ex. yes/no; true/false; reliability scores usually above 0.70
- Cronbach's alpha: compare each item in the scale to other items in the scale – correlation matrix; comparing in a scale to other items and see if correlated; good alpha is 0.7-0.9 (0.9 could be either be excellent or indicate duplication of items)
Equivalence:
- the degree of similarity between alternative forms of an instrument or between multiple raters/observers using an instrument (want to be sure we are getting the same score no matter the process)
- most relevant for structured observations
- assessed by comparing observations or ratings of 2 or more observers (inter-observer/inter-rater reliability)
- numerous formula and assessment methods
Validity:
- the degree to which an instrument measures what it is supposed to measure
- are we measuring what we are believing we are measuring
- 4 Aspects:
- Face Validity: have a group similar to your participants use the measure and get their feedback (did they think it made sense? were they confused?)
- Content Validity: want to make sure that the instrument being made is correct and the content it is giving is correct
- Criterion-related Validity: agreement between responses on measure and actual behaviour (does the person actually do what they said they do on the measure?)
- concurrent = two different measures at the same time (same construct) and trying to understand if measurements have the same constructs
- predictive = measure it now and at a later time and see if you are getting the same thing
- Construct Validity: How well the tool measure the theory behind it? Does the tool validate the body of evidence underlying the tool development?
- 1. Hypothesis testing (a little more complex)
- 2. Convergent Validity (use an already validated tool that measures the same construct and give both to participants)
- 3. Divergent Validity (try to see if you constructs are different than other similar constructs; use another measure that measures something similar but different and see if you get similar or different responses); if we are getting at the same thing then something is wrong as that is not what we are wanting to get at
- 4. Contrasted Group Approach (give tool to 2 groups and see if responses match expectations)
- 5. Factor Analysis (statistical way to measure the construct validity)
Measurement Error:
- important so that you are measuring in the right way (quality measurement for quality data)
- Errors of Measurement: difference in scores that is due to error rather than due to differences between participants
- obtained score: an actual data value for a participant (ex. anxiety scale score)
- true score: the score that would be obtained with an infallible measure; get from a perfect measure - hardly happens since it is hard to get a perfect score
- error: the error of measurement, caused by factors that distort measurement
- obtained score = true score +- Error
Factors that Contribute to Errors of Measurement:
- Situational contaminants
- Transitory personal factors
- Response-set biases
- Administration variations
- Problems with instrument clarity
- Item sampling
- Instrument format
Sources of Error: (control for what you can and be monitoring)
- Random or Chance Error: harder to control
- human condition (measuring concentration but participant is tired or hungry)
- measurement variation (measuring in blood pressure room is colder or warmer)
- data entry (human error – enter wrong numbers)
- Systemic Error:
- most participants are unintentionally similar (all same SES)
- socially desirable responses (telling you what they think you want to hear)
- equipment malfunction (ex. pumps)
Summary of Reliability/Validity:
- Assess the appropriateness of the type of validity and reliability selected for a particular research --> splits into consistency (reliability) and accuracy (validity)
- whole summary is slide 39 (go review)
-