Please enable JavaScript.
Coggle requires JavaScript to display documents.
SOCR8204, Pre-lecture 2: Survey data analysis - testing for associations,β¦
SOCR8204
Pre-lecture 1
Qualitative vs Quantitative Data
- Qualitative data can be types.
- Qualitative data is numeric or count
Definitions:
- Data is usually numeric and is collected through observations;
- Survey: investigation about characteristics of a given population by collecting data
- Variable: characteristic of a unit observed that can have more than one of a set of values.
- Parameter: study of variable values that forms a set for system of operations.
Dimensions
Dimension 1 - Coverage and representativeness
- Population: the complete collection of all units that one is interested in
- Sample: the subset of the population that is observed by data collectors.
- Sampling frame: the set of the target population members that have a chance to be selected.
- Probability sampling:
- each unit of population has a chance being selected
- in every unit have the same chance then it is a simple random sample
- under cluster sampling, geographic area is randomly identified first before selecting individuals from the frame.
- Census: where sample is the same as the population.
Dimension 2 - Collection methodology
- Primary data collection
- face to face interview
- focus groups
- mail back
- telephone interview
- online interview
- Textual data
- Administrative data: data already available.
Dimension 3 - Measurement
- Data type:
- counitnuous (height, income)
- count (number of production)
- duration (number of months of unemployment)
- binary (male/female)
- categorical (coalition/labor/greens)
- All of the above can be:
- objective
- subjective (can be confirmed)
- attitudinal (type of objective)
Dimension 4 - Repetition
- Cross sectional survey: sample taken from one at one point in time.
- Repeated cross sectional survey: same questions asked more than once of the same population. Sample selected afresh each time.
- Longitudinal survey: survey conducted that tracks information through time.
- Qualitative, quantitative and admin data can be cross-sectional or longitudinal.
Census:
- All people in Australia apart from diplomats and famly;
- Visitors to Australia is counted as separate identity;
- Australian residents out of the country are not counted;
- Self enumerated completed by pen or online;
- Less census collectors now;
- Personal forms and privacy envolop for non-private dwelling;
- Special collection for indigenous group, homeless, foreign lanaguage and people travel on the night.
Census: from count to population estimate I
- Main output: count population and characteristics on the night of census.
- available for place of enumeration and area of usual residence
- available for place of work
- Census counts differ from 'census year population estimates':
- undercount (those not counted)
- overcount (those were counted more than once)
- Australian residents temporarily overseas
- Corrections needed for partial response:
- imputed for age, sex, marital status and SA1 or UR
- 'not stated' for all other census quetsions
Census: from count to population estimate II
- Post-enumeration survey (PES)
- used to calculate the net undercount
- survey of over 40,000 private dwellings by trained interviewers
- Respondents asked if they were included on a Census form
- Personal information is matched to any corresponding census forms
- separate undercount by state/territory, sex, marital status, age, indigenous status and country of birth
- high undercounts for people with high mobility and young people filling census the first time.
- Undercount can be affected by country of birth cohort
From census to sample
- One of the great breakthroughs in social science in the 20th century is the realisation that precise estimates from a fraction of the population
- To make inference about the population:
- sample must be representative of total population and sub-groups of interest
- each individual in the sample can be made to represent a different number of people
Key cross-sectional surveys in Australia
- Labour force survey: unemployment rate, labour force participation rate, unemployment
- Australian health survey: estimates for prevalence of particular diseases and health behaviours
- Australian Early Development Census (AEDC): estimates of developmental vulnerability for children in their first year of full-time schooling
- Programme of Interational Student Assesment (PISA): cross-national comparisons of literacy/numeracy of 15 year olds
- Quality Indicators for Learning and Teaching:
- studnet experience survey, graduate destination suryve
- which universities are doing better or worse
Longitudinal Surveys
- Longitudinal survey is a sample survey where individuals, households, business are more than once with information tracked through time.
- Three types:
- cohort studies: study a group experiencing some event in a selected time period, and studying them at intervals through time.
- panel studies: select a representative sample of individuals at a particular point in time and follow them through time.
- administrative collections or linked surveys: track individuals through time based on adminitrative data. Can involve linking information on individuals from before and/or after a croos-sectional survey.
- Advantanges:
- allows analysis of factors associated with change through time
- allows one to control for unobserved, time-invariant characteristics
- information available about the past without recourse to recall
- Disadvantages:
- sample can become unrepresentative through time
- cohort studies not necessarily generalisable
Household Income and Labour Dynamics in Australia (HILDA)
- Household-based panel study which began in 2001
- Collects information about economic and subjective well-being, labour market dynamics and family dynamics.
- Interviews are conducted annually with all adult members of each household.
- The funding has been guaranteed for sixteen waves, though the survey is designed to continue for longer than this.
International Panel Studies
- HILDA intentionally fits within an international set of surveys on income and labour dynamics
- Made available through the Cross-National Equivalent File
Longitudinal Surveys of Australian Youth (LSAY)
- Focuses on youth outcomes and transitions, following successive cohorts of 15 years olds
- Based on cross-sectional sample from the Programme for International Student Assessment (PISA)
- Sampling undertaken at the school level with a random sample of schools selected and a random sample of students from each selected school
- High rate of sample attrition
Longitudinal Study of Australia Children (LSAC)
- The LSAC aims to provide a database for a comprehensive understanding of children's development in Australia's current social, economic and cultural environment.
- LSAC was constructed around two corhorts.
Longitudinal Study of Indigenous Children (LSIC)
- The LSIC is first large scale longitudinal survey in Australia to focus on the development of idigenous children
- Four key research questions:
- Two cohorts: 960 babies, 727 children
- Sample attrition reasonably low considering highly mobile population
Qualitative data methods
- Involves analysis of data such as words, pictures or objects
- Participant observation:
- Structured internview
- Semi-strucutred interview/unstructured interview
- Focus groups
- Key informant, anlaysis of docments and materials.
Strengths and limitations
- Strengths:
- provides a rich description of phenomenon, including why and how.
- useful for generating hypothesis
- can be reasonable cost effective
- Policy insights:
- how a policy or program is actually implemented on the ground,
- how participants understand it,
- how it plays out for individuals in specific cultural contexts
- Limitions:
- implications and conclusions are based on particular context and not always generalisable
- less useful for making out-of-sample predictions
Mixed methods approach to program evaluations
- Motivations
- a number of important outcomes are difficult to reduce to a single number
- qualitative research can identify effects that weren't predicted a priori
- qualitative methods can identify the process, not just the outcomes from an intervention
- some subjects are too small or geographically dispersed to get a large enough sample from
- they can be better suited to subjects with relatively low literacy levels
- Mixed methods approaches
- parallel mixed methods: qualitative or quantitative research done separately with results compared at the end
- sequential or iterative mixed methodsοΌ interaction between qualitative and quantitative research
- Quasi-mixed methods: culturally informed quantitative research/empirically informed qualitative research
Qualitaitve methods and findings
- methods
- qualitative, exploratory
- semi-structured, in-depth interviews
- 34 former APS employees
- 13 departements and agencies
- Findings:
- overselling the APS
- politics and policy
- career and supervision
- racism and reponse
- Being undervalued
Administrative Data
- Government department and agencies build up administrative data collections during day to day activities
- Routinely gather information when registering people or carring out transactions
- While not collected for research purpose, it cna be used for answer research questions.
strength and limitations
- Strengths
- admin data collections already exist
- admin data are frequently collected inthe same way over many years.
- use historical inforamtion to compare the same data
- admin data collections tend to cover the whole population
- the collection process is not intrusive
- reduces respondent burden
- Limitations
- can be complicated to analyse
- burden transferred to service providers
- hard to access, including metadata
- privacy concerns
- data not specific to researchqusetions
- information not available on those who don't use the service
- Data linkage types
- admin to admin
- survey to survey
- admin to survey
- Mthods: deterministic vs probabilistic
-
-
Pre-intensive Week 4
Time Series
Time Series
- Series of data points (economic data in the lecture) in time order, usually a sequence taken at equally spaced points in time. e.g. yearly, quarterly, monthly, daily, or higher frequency.
- E.g.:
- GDP
- unemployment and employment rates (monthly labour force)
- retail sales (monthly)
- consumer prices (quarterly CPI
- financial data such as stock and bond prices (daily, intradaily)
- population (quarterly demographic statistics)
- COVID-19 statistics (daily)
Software for Time Series Data
Time Series Notation:
- T is the number time series observations
- Time series regression models can be used for dynamic causal effects or faorecasting
- First difference: Ξπ=ππ‘βπ(π‘β1)
- Percentage change: 100ΓΞπ/π(π‘β1) =100Γ((ππ‘βπ(π‘β1) ))/π(π‘β1)
- First log difference: Ξπππ= ln(ππ‘ )βln(π(π‘β1) )=lnβ‘(ππ‘/π(π‘β1) )
Trends:
- Trend: a persistent long-term movement of a variable over time
- Deterministic trend: a non-random function of time
- can be removed by de-trending the data
- Stochastic trend: random and varies over time
- Problems caused by stochastic trends
- autoregressive coefficients are biased toward zero
- non-normal distributions of t-statistics
- spurious regression: stochastic trends can lead two time series to appear related when they are not.
- A spurious regression would arise as GDP and debt levels look correlated and come out as highly significant in an OLS regression.
- But it is questionable that higher debt levels cause higher GDP level.
Autocorrelation
- In time series data, the value of Yt = in one period tends to be correlated with its value in the next year
- The series is said to be persistent
- More formally, if a variable Yt is correlated with itself over time, then it is said to be autocorrelated or serially correlated.
Autocorrelation
- The error term Ξ΅t is also frequently autocorrelated in time series data as it consists of time-varying factors that are determinants of Yt but are not included as regressors
- some of these omitted factors might be autocorrelated
- When regression errors are autocorrelated, our Multiple Linear Regression assumption is violated
Intensive Week
Day 1
Stages of Reserch Process
- Literature Review
- Concepts and theories
- Research questions
- Sampling cases
- Data collection
- Data analysis
- Writing up
How to narrow a topic into Research Question
- Examine the literature
- Talk over ideas with others
- Apply a specific context
- Define the aim or desired outcome
Good research questions:
- Exploratory
- Descriptive
- Explanatory
- Causal
-
Social Theory: A system of interconnected ideas that condenses and organises knowledge about the social world.
Types of evaluation:
- Formative evaluations (ε½’ζζ§θ―δΌ°): examine a program's development and assist in improve its structure and implementation
- Summative evaluation (ζ»η»ζ§θ―δΌ°οΌ: assess whether objectives were achieved, but may look for a broader array of outcomes.
- Process evaluation: used to measure the activities of the program, program quality and who it is reaching.
- Outcome evaluation: used to measure the impact of the program on the target group and/or more broadly. Mostly at the end of the project.
Correlation/association and causation:
- Correlation does not equal to causation
Evaluation methods:
- Before/after studies
- Experimental evaluations (randomised controlled trials)
- Qualitative evaluations
Quantitative program evaluation - overview and problems
- Observable outcome indicator for an individual (Yi)
- Particular program (or treatment) aims to bring about improvement in outcome
- Individuals is assumed to have a different outcome if they received treatment (), than if did not (οΌ
- Gains defined as
- Missing data: a person can not be treated or not treated at the same time.
- Only solutions: compare same people different times or different people the same time.
Quantitative program evaluation - selection bias
- Cannot assume that in the absence of the program.
- Selection bias: individuals in treatment group do not necessarily have the same mean (absence of treatment) outcome compared to those in the control group.
- Differences in outcomes can be due to unobservable characteristics.
Quantitative program evaluation - Inappropriate Control Group
- Reverse causality
- Unobserved heterogeneity
- Example of self-selection
The principal of random assignment
- Allocation into treatment/control groups through 'toss of a coin'
- Key requirement for random assignment, assignment is statistically independent.
- Consequences of random assignment
- The sample is representative of the population
- Same proportion for young, old, etc.
- Only systematic difference is the treatment.
- Estimation of treatmnet effectis is straightforward
- actual average outcome of control group = counterfactual average outcome in the treatment group
- counterfactual average outcome in the treatment group = actual average outcome of control group
Opportunities to randomise
- Basic elements of an intervention which can be randomised: access, timing, encouragement
- Opportunities to design a trial: new program, expansion, change in program, oversubscription for existing program
Field experiment
- preparation
- design
- pre-analysis
- conducting the trial
- analysis
- conclusions
The EAST principle
- Make it easy
- Make it attractive
- Make it social
- Make it timely
-
Intensive Week - Day 2
RCT
Field Trials
- Field experiments are 'randomised studies that are conducted in real world settings
- Lab and field experiments are two ends of a sppectrum with four criteria to distinguish:
- authenticity
- participants
- context
- outcomes
Over-view of field experiment
- preparation
- design
- pre-analysis
- conducting the trial
- analysis
- conclusions
Population and samples
- popuation
- sample
- sample frame: the set of target population members that have a chance to be selected into survey sample
- probability sampling:
- each unit has chance to be selected
- each sample has the same chance, then it is a simple random sample
- under cluster sampling, is to first randomly identify geographic areas.
- stratified sampling is to sample from the sub-groups
Central limit theorem and sampling variation
- When independent random variables are added, their sum tends toward a normal distribution, even if the original variables are not normally distributed
- Standard error of the mean (SEM) is the stanard deviation of the sample mean's estimate of a population mean.
Inference
- Hypothesis test:
- if a null hypothesis should be accepted or rejected.
- statistical significanceοΌ
- a result in statistics that is unlikely to occur by chance
- Confidence interval:
- a range of values for a variable of interest constructed so that the range has a specified probability of including the true value of the variable. (95% confident that mean value is within the range)
Power calculation
- Power analysis can be used to calculate the minimum sample size required that can reasonably likely to detect an effect of a given size
- power analysis can also be used to calculate the minimum effect size that is likely to be detected in a study using a given sample size
- alpha: the probability of a type-I error (finding difference when difference not exist). Mostly use 5%, indicating a 5 % chance that a significant difference is actually due to chance and is not a true difference.
- beta: the proability of a type-II error (not detecting a difference when one actually exists). Power = 1 - B. Mostly 20%, indicating that 20 % chance that a significant difference is missed.
Sample size formula for difference in means:
General sample size needs when outcome is binary:
Sample stratification and heterogeneous treatment effects
- Standard randomised design assumes that the effect of the treatment is consistent across the population. This assumption may not hold:
- particular groups of the population might respond differently to the treatment
- the treatment might only be effective for those at certain points on the distribution
- background characteristics might affect whether someone agrees to the treatment or drops out of the sample
- Sample size calculations do not guarantee that we will be able to analyse these effects:
- power calculations are used for each sub-group to obtain stratified sample
- more complicated if we can only measure sub-groups ex post.
Randomisation at the group level
- spillover effects: a key assumption for a randomised evaluation is that the outcome of one person does not depend on the group (treatment/control) to which other people they interact with are allocated.
- physical spillovers
- behavioural spillovers
- informational (social learning)
- marketwide or general equilibrium
- Solution is to randomise at the institutional/group level, but this effectively reduces the sample size
- there maybe something about the groups that are idiosyncratic
- standard errors are clustered
RCT Analysis - Itention to treat (ITT) effect
- The ITT estimates the difference in outcomes for all those assigned to the treatment compared to the outcome of the those assigned to the comparison group
- what happens to the average person given access to the program
- It does not measure the effect of the treatment program itself
- Can be measured using a regression framework:
Covariates in ITT
- Covariates allow us to compare the differences in outcomes in the treatment and control group whilst holding observable characteristics constant:
- If sample size was extremely large, then there would be no unobservable differences between the two groups
- but includigng covariates that help predict the outcome variable reduces the unexplained variance, making the estimates more 'precise'
- should effect the standard errors, much more than the coefficients
- very important to include only covariates that are measured before assignment, that don't affect assignment, or that don't change through time (e.g. gender)
- For example, you are running an experiment to see how corn plants tolerate drought. Level of drought is the actual βtreatmentβ, but it isnβt the only factor that affects how plants perform: size is a known factor that affects tolerance levels, so you would run plant size as a covariate
Clustered standard errors in ITT
- Clustered Standard Errors(CSEs) happen when some observations in a data set are related to each other. This correlation occurs when an individual trait, like ability or socioeconomic background, is identical or similar for groups of observations within clusters.
- when randomising at the group level or when analysing data with significant correlation across groups, need to control for clustering in outcome data
- error term is correlated across individuals
- Clustered SE increase the confidence interval, meaining we are less likely to conclude that differences are significant
Average Effect on the compliers (Local Average Treatment Effect)
Assumptions for measuring LATE
- One-sided non-compliance
- Two-sided non-compliance
- Equivalent to a two-stage least squares (2SLS) estimate, where treatment is the first stage.
- The LATE is not the same as the ATE - it does not measure the effect of the treatment on Never-Takers
- LATE is almost always necessary for 'randomised promotion' or 'encouragement design'
Issues with random assignment
- Randomisation of treatment is often infeasible
- Subjects may self-select into or out of treatment
- Program managers may mess up random assignment
- Randomisation of eligibility is often all that is feasible
- Sometimes all individuals/units need to receive the treatment
- Spillover effects: treatment may have unintended consequences on the control group
- Sample attrition: there may be difficulties in keeping track of individuals in the control or treatment group forthe length of time over which the intervention is expected to have an effect.
- Non-response or recall bias: not all individuals may have the information on outcomes or observables or they may make errors in recall.
- Ethics: randomised controlled trials necessitate excluding individuals from treatment
- Publication bias: those studies that find a positive effect are more likely to be published.
- Expense: rigorous evaluation can be expensive.
-
Intensive Day - 5
Ethics
- Main issues:
- permanent harm
- not voluntary
- no informed consent
- deception
Why ethics:
- a set of principles is required to help researchers deal with competing demands and interests
- application of principles achieved through
- institutional regulation
- research training
- interaction with colleagues and peers
- practical experience of conducting research
-
Definition
Cost-benefit analysis (CBA) estimates and aggregates the monetary value of the benefits and costs to society of projects/interventsion/policies to establish whether they are worthwhile
Calculating value for money
- Two main VFM estimation methodologies
- cost effectiveness analysis (CE): used for measuring programmes that aim to achieve the same goal, or for when the main focus is on the relative effectiveness or VfM of two programmes.
- cost benefit analysis (CBA): attempt to measure whether a programmes is beneficial in an absolute sense.
Steps
Six steps in CE and CBA
- identify key outcomes and how they are to be defined and measured
- gather and analyse cost data
- examine causality and calculate the net impact of the program
- discount effects and costs
- analyse the distribution of effects
- combine costs and effectiveness
- It is sometimes necessary in CBA to evaluate the benefit of saving human lives
Meta Analysis
- A meta-analysis combines the results from multiple studies in an effort to increase power, improve estimates of the size of the effect and/or to resolve uncertainty when reports disagree.
- It provides a weighted average of the included study results
- Advantages:
- results can be generalised to a larger population
- the precision and accuracy of estimates can be improved as more data is used.
- inconsistentcy of results acorss studies can be quantified and analyzed. e.g. does inconsistency arise from sampling error, or are study results influenced by between study heterogeneity.
- hypothesis testing can applied on summary estimates.
- moderators can be included to explain variation between studies
the presence of publication bias can be investigated. :
Non-market valuation
- many policies and projects involve inputs and outputs for which there are no formal markets or market prices. This is particularly true for policies/projects with environmental and/or social/healh impacts.
- Non-market impacts are just as relevant to societal welfare as market impacts
- it is important, therefore, that non-market impacts are included in CBA wherever possible
- To do this, it is necessary to value the impacts in monetary terms via a process known as 'non-market valuation'.
Measuring non-market values: approaches
- The revealed preference approach relies on observations about people's behaviour in markets that are someway related to the good or service under consideration
- The stated preference approach uses surveys to question how respondents value that good or service.
Common stated preference technique:
- contingent valuation
- choice modelling
Contingent valuation is a survey-based economic technique for the valuation of non-market resources, such as environmental preservation or the impact of contamination
Choice modelling attempts to model the decision process of an individual or segment via revealed preferences or stated preferences made in a particular context or contexts.
Relative new, hybrid, non-market valuation technique:
- life satisfaction approach
Common revealed preference techniques:
- hedonic pricing
- travel cost
The hedonic price method uses the value of a surrogate good or service to measure the implicit price of a non-market good.
For example, house prices can be used to provide a value of particular environmental attributes. The worsest environmental character has a negative impact on house price.
Issues with random assignment
- Randomisation of treatment is often infeasible
- subjects may self-select into or out of treatment
- program manager may mess up random assignment
- randomisation of eligibility is often all that is feasible
- sometimes all individuals/units need to receive the treatment
Quantitative program evaluation issues
- spillover effects: treatment may have unintended consequences on the control group
- Sample attrition: there may be difficulties in keeping track of individuals in the control or treatment group for the length of time over which the intervention is expected to have an effect
- non-response or recall bias: not all individuals may have the information on outcomes or observables or they may make errors in recall.
- ethics: randomised controlled trials necessitate excluding individuals from treatment
- external validity: can results from a rigorously evaluated intervention be applied in other settings? Is the intervention cost effective?
- publichation bias: those studies that find a positive effect are more likely to be published
- expense: rigorous evaluation can be expensive. Is it worth doing a less rigorous evluation than none at all?
Non-experimental, Quasi-experimental, RCT
-
-