Chapter 7: Scale Reliability and Validity
Theory of Measurement
Reliability
Validity
Levels of Measurement
Measurement Error
Definition: Systematic Assignment of Numbers or Labels to Objects and Their Attributes
Ordinal Scale
Interval Scale
Ratio Scale
Nominal Scale
Classification into Categories (e.g., gender, ethnicity)
Rank Ordering of Categories (e.g., class ranking)
Equal Intervals Between Values, No True Zero (e.g., temperature in Celsius)
True Zero Point and Equal Intervals (e.g., weight, height)
Random Error
Systematic Error
Consistent, Predictable Error (e.g., a scale that always adds 5 pounds)
Unpredictable, Unsystematic Error (e.g., fluctuations in a person's mood)
Sources of Unreliable Observations
Strategies for Improving Reliability
Definition: Consistency or Dependability of a Measure
Methods of Estimating Reliability
Example: Weight Scale vs. Guessing Weight
Subjectivity in Qualitative Measures
Imprecise or Ambiguous Questions
Lack of Familiarity or Interest in Topics
Reducing Subjectivity
Asking Clear and Relevant Questions
Simplifying Wording
Test-retest Reliability
Split-half Reliability
Inter-rater Reliability
Internal Consistency Reliability
Categorical Measures: Percentage of Agreement
Interval/Ratio Measures: Correlation Between Raters
Consistency Between Independent Raters
Consistency Over Time
Importance of Time Interval
Consistency Between Two Halves of a Measure
Tendency to Overestimate Reliability of Longer Instruments
Average Inter-item Correlation
Cronbach’s Alpha: Formula and Calculation
Consistency Between Different Items of the Same Construct
Definition: Adequacy of a Measure in Representing the Construct
Types of Validity
Example: Compassion vs. Empathy
Translational Validity
Criterion-related Validity
Methods of Assessing Validity
Face Validity
Content Validity
Reasonableness of an Indicator as a Measure
Match Between Scale Items and Content Domain
Discriminant Validity
Concurrent Validity
Convergent Validity
Predictive Validity
Closeness of Measure to the Construct
Distinction Between Different Constructs
Agreement with Other Measures of the Same Construct
Ability to Predict Related Outcomes
Theoretical Approaches
Empirical Approaches
Panel of Expert Judges
Q-sort Technique
Experts rate each item for relevance and representation
Sorting items into categories that represent different levels of the construct
Factor Analysis
Multi-trait Multi-method (MTMM) Approach
Correlational Analysis
Examining the relationship between the measure and other relevant measures
Principal Components Analysis
Factor Loadings for Convergent and Discriminant Validity
Rotated Factor Matrix
Identifying the underlying structure of the data
Assessing which items load on which factors
Clarifying the structure by rotating the factors
Assessing the validity of a measure by comparing it with multiple traits and methods
Chapter 8: Sampling
The Sampling Process
Non-Probability Sampling
Types
Statistics of Sampling
Statistics of Sampling: Key Terms
Definition: The process of selecting a subset of individuals from a population to estimate characteristics of the whole population.
Steps
Specify a sampling frame.
Specify a sampling method.
Define the population.
Determine the sample size.
Implement the sampling plan.
Collect data from the sample.
Review the sampling process.
Systematic Sampling:
Stratified Sampling:
Simple Random Sampling:
Cluster Sampling:
Every subset of the population has an equal chance of being selected.
Example: Using a random number generator to select sample units.
Selecting every kth element from an ordered sampling frame.
Example: Choosing every 10th name from an alphabetical list.
Dividing the population into strata and randomly sampling from each stratum.
Example: Sampling different age groups separately.
Dividing the population into clusters and randomly sampling entire clusters.
Example: Selecting entire schools rather than individual students.
Definition: A sampling technique where some elements of the population have no chance of being selected or the probability of selection cannot be accurately determined.
Types
Quota Sampling:
Purposive (Judgmental) Sampling:
Convenience Sampling:
Snowball Sampling:
Selecting samples that are easiest to access.
Example: Surveying people at a mall.
Ensuring the sample represents certain characteristics of the population.
Example: Ensuring equal numbers of males and females in a study.
Selecting based on the researcher’s judgment about who is most useful.
Example: Choosing experts in a field for a study.
Existing study subjects recruit future subjects from among their acquaintances.
Example: Studying a hidden population like drug users.
Population Parameter: A value that describes a characteristic of an entire population.
Sample Statistic: A value that describes a characteristic of a sample.
Sampling Error: The difference between a population parameter and a sample statistic.
Sampling Distribution: The distribution of a sample statistic over many samples.
Standard Error: The standard deviation of a sampling distribution.
Confidence Interval: A range of values that is likely to contain the population parameter.
Sample Variance (s²):
Standard Error (SE):
Sample Mean (x̄):
Confidence Intervals (CI):
Margin of Error (MoE):
The average value of the sampled observations.
Used to estimate the population mean (μ).
The measure of variability in the sampled observations.
Used to estimate the population variance (σ²).
The standard deviation of the sampling distribution of the sample mean.
SE = s / √n, where s is the sample standard deviation and n is the sample size.
A range of values that is likely to contain the population parameter.
Example: 95% CI for the mean.
The maximum expected difference between the sample statistic and the population parameter.
MoE = Z * SE, where Z is the z-score corresponding to the confidence level.