Please enable JavaScript.
Coggle requires JavaScript to display documents.
Research Methods chapter 13 (threats to the validity of meta-analysis…
Research Methods chapter 13
generalizing from single versus multiple studies
multiple studies usually have greater samples of persons, settings, times, treatments, and outcome measures
the same researcher can build off his own previous studies
a researcher can find different studies that have a wide variety of aspects to them increasing diversity
multistudy programs of research
allow the researcher to pick the aspects from each study they find relevant
box score or vote counting: counting the wins, losses or ties that a treatment has at being effective and adding them up for comparison
phased models of increasingly generalizable studies
i.e. FDA. these programs move in phases that start from simple research to more complex as the program develops and gains momentum.
surface similarity in the use of human cancer explanation can be seen
ruling out irrelevancies in deliberately making patient characteristics diverse, such as type of cancer, age, and gender of patient in early dosage-toxicity trials
making discriminations about which kinds of cancers are most and least affected
interpolation and extrapolation in varying drug doses in initial trials to see how both toxicity and clinical response might vary over that continuum
causal explanation in developing models of how a drug acts on human cancer cells so that potency of that action can be increased
directed programs of experiments
the aim is to systematically investigate over many experiments the explanatory variables (moderators, mediators, constructs) that may account for an effect, gradually refining generalizations
narrative reviews of existing research
because research is expensive, people have started reviewing previous research to clues about phenomenon
narrative views of experiments
describe the studies without attempting to do a quantitative analysis
narrative reviews combining experimental and non-experimental research
may include not only field studies, but also surveys, animal studies, and basic lab studies to prove generalizability
these studies are useful for when manipulation of a human or animal would be unethical
problems with narrative reviews
researchers have to comb through so many studies it can be difficult to keep all of the measures in order
narrative reviews traditionally rely on box score summaries of results generated from the significant tests of study outcomes
narrative reviews are not very precise in their descriptions of study results
5-10% produce non significant results
matters become even more complex when a narrative reviewer tries to examine relationships among outcomes and potential moderators
quantitative reviews of existing literature
accumulating effect sizes over studies became termed meta-analysis
this became popular because the same measures are rarely used in different studies
the basic meta-analysis
quantitative and qualitative reviews both go through a series of problem formulation, data collection, data evaluation, analysis and interpretation, and public presentation of results
identifying the problem and doing the literature review
little about the problem formulation in meta-analysis is different from question formulation in primary studies or narrative reviews. the aim is to develop a clear research question and, based on this, a tentative framework of criteria for including studies in the meta-analysis
many studies can be located from computer data-basis, by inspecting reference lists of previous reviews, by scanning tables of contents of recent journals, by identifying registers of past and ongoing trials, and by contacting the "invisible college" for colleagues with special interests in the research question
some studies will be hard to locate and are often called fugitive literature. these are unpublished dissertations and master's theses, final reports of grants or contracts, convention papers, technical reports, and studies subject to the file draw problem - papers rejected for publication that are then relegated to the file draw
coding of studies
individual's codes should first and foremost, reflect the researchers hypothesis.
some coding require detailed instructions and seasoned judgements, as when rating the reactivity of a measure or categorizing the theoretical orientation of a variable. in other words it should follow specific rules, should be tested for inter-rater reliability, and should be revised until adequate reliability is obtained.
coding protocols should be developed with an eye toward how the data base will be entered for subsequent computer analysis and whether initial coding will be done directly into computer files
computing effect sizes
the standardization mean difference statistic
the odds ratio
analyzing meta-analytic data
can use univariate or multivariate techniques
the desirability of weighting effect size estimates by a function of study sample size
individual studies are often weighted prior to averaging, which can yield substantially different answers than unweighted analysis
the use of tests for homogeneity of effect sizes
this tests whether a set of observed effect sizes vary only as much as would be expected due to sampling error-the part of the difference between the population effect size and the sample estimate of that effect size that is due to the fact that only a sample of observations from the population were observed.
the hierarchical nature of meta-analytic data
a multilevel model that can be used to combine study level data within individual level data for use when some studies report summary statistics, but individual level data are available for other studies
the dependency of effect sizes within studies
can arise when a study has multiple outcome measures, when it reports results on those outcomes at more than one time point, or when it compares several interventions with a common control condition on just one measure
the presence of publication bias
the problem results because many reviewers are prejudiced against recommending the publication of studies with nonsignificant findings
interpreting and presenting results
subject to bias from the person interpreting or publishing
we never randomly assign studies to the person, setting, time, cause, and effect category we analyze
rarely use quasi-experimental design techniques
meta-analysis and the five principles of generalized causal inference
surface similarity: the emphasis is on assessing the match between research operations and the prototypical features of the targets of generalization.
multiple studies can usually represent many more constructs than one study can
reviews greatly increase the chances of finding studies with operations that reflect constructs that may be of particular policy or research interest
some studies may not exist for a target population. in this case, one of the main benefits of reviews is that they highlight important constructs that a literature has mostly ignored so that future research can be aimed at them.
ruling out irrelevancies
identify certain limits to the generalization of a causal claim
finding similar things in all the studies that are irrelevant
making discriminations
here the strategy is to demonstrate that an inference holds only for one construct as specified, not for some alternative or respecified construct
as or more important, however, is the role of such discriminations in clarifying the range and boundaries of a causal relationship
interpolation and extrapolation
intrapolation: find the value between two variables
extrapolation: predicting what will happen based on the current trends
causal explanation
meta-analysis makes it easier for the researcher to break down persons, settings, treatments, and outcomes into their component parts in order to identify their causal relevant components
meta-analysis can use multiple regression equations to identify redundancy among variables that moderate outcome, helping to narrow the explanation and to better access the magnitude of effect associated with different predictors
full explanation also requires analysis of the micromediating causal processes that takes place after a cause has been varied and before an effect has occurred
climate: that high expectations lead to a more differentiated feedback about accuracy of student responses
feedback: that high expectations lead to more differentiated feedback about the accuracy of student responses
input: that teacher will teach more, and more difficult, material to high expectancy students
output: that teachers will give high expectancy students more opportunity to respond
the statistical methods that are so useful for meditational modeling in primary studies-in the previous chapter have not been adapted to meta-analysis
discussion of meta-analysis
research synthesis: reviews would still seem worthwhile, power problems are not
threats to the validity of meta-analysis
unreliability in primary studies: it attenuates effect sizes from primary studies. attenuation corrections can correct estimates during the meta-analysis
restriction of range in primary studies: restriction of range in the outcome variable can attenuate effect sizes in a primary study, and these attenuated effect sizes will carry forward to the meta-analysis. corrections can be made if valid estimates of population range or variance are available
missing effect sizes in primary studies: missing effect sizes occur when researchers in primary studies (1) fail to report details about statistically nonsignificant findings or (2) mention using an outcome measure but never discuss it in the results.
unreliability of coding in meta-analysis: the variables used in meta-analytic coding can be unreliable and can attenuate meta-analytic relationships, particularly given common practice of coding only one item to represent a construct.
capitalizing on chance in meta-analysis: meta-analyses conduct many tests to probe relationships between effect size and various predictors. to reduce capitalization on chance, the researcher can adjust error rates using Bonferroni corrections or use multivariate procedures such as regression
biased effect size sampling: bias occurs when a meta-analyst codes only some of the plausibly relevant effect sizes from a study if the omitted ones differ from the coded ones on average.
publication bias: published studies may be a biased sample of all studies ever done and may also overestimate effects. minimizing this bias requires (1) strenuous attempts to find unpublished studies, (2) testing separate effect size estimates for published and unpublished studies, and (3) assessing potential consequences of this bias using one of the several methods now available for doing so
bias in computing effect sizes: primary studies often fail to provide the data needed to compute an effect size, and approximations to these effect size estimate vary in accuracy
lack of statistical independence among effect sizes: effect size estimates may lack statistical independence because (1) several effect size estimates are calculated on the same respondents using different measure; (2) several effect sizes compare different interventions with a common comparison group; (3) effect sizes are calculated on different samples in one study; and (4) different studies are conducted by the same research team over time.
failure to weight study level effect sizes proportional to their precision: studies with larger sample sizes should receive more weight in order to increase the precision of the average over studies. weighting by the inverse of effect size sampling error (a function of sample size) minimize the variance of the weighted average effect size. weighting by sample size does not have this latter advantage but improves precision considerably compared with no weighting at all.
inaccurate homogeneity tests: homogeneity tests influence several decisions in meta-analysis, such as the choice between fixed and random effects models or the decision to continue searching for moderators of effect size. when primary studies have very small sample sizes, the tests may have lower power.
under-justified use of fixed effect models: fixed effects models are often used, but not as often well justified in meta-analysis. however, fixed effect models can sometimes be justified despite rejection of homogeneity, for example, if adding predictors to a regression equation accounts for remaining heterogeneity or if the researcher has an interest in the robustness of observed studies to sampling error
lack of statistical power: all other things being equal, statistical power is higher in meta-analysis than in primary studies.
threats to inferences about the causal relationship between treatment and outcome
failure to assign to primary treatments at random: many meta-analysis examine questions about treatment effect. meta-analysts may have confidence in causal inferences about treatment contrasts to which participants were randomly assigned in the primary studies-certainly no less confidence than in the primary studies themselves-subject to the usual caveat such s low and nondifferential attrition that usually apply to those studies.
primary study attrition: attrition is routine and differential in some areas
moderator variable confounding: even when participants were randomly assigned to treatment contrasts that are the primary focus of a meta-analysis, neither participants nor studies are randomly assigned to potential moderators of that effect, such as publication status, measurement techniques, or study setting
threats to inferences about the constructs represented in meta-analysis
underrepresentation of prototypical attributes: studies often do not contain representations of all of the prototypical elements of a target construct
mono-operation bias: this threat and the next draw attention to the typically poor measurement properties of the vast majority of meta-analytic coding
monomethod bias: similarly, the vast majority of construct measurement in meta-analysis relies on a single method-the use of one coder who exercises largely independent judgement in assessing the properties of study
rater drift: this threat is similar to treatment sensitive factorial structure, and refers to changes in consecutive ratings by the same rater over time
reactivity effect: parallel to the several reactivity and expectancy effects, refers to certain extraneous forces in the meta-analytic coding protocol
confounding constructs with levels of constructs: just as in primary studies, meta-analysts may use construct labels that fail to describe the limited levels of the construct that were actually studied
confounding constructs with other study characteristics: in meta-analysis, constructs of one kind (treatment constructs) are often confounded with those of another kind (settings, times, outcomes, persons)
misspecification of causal mediating relationships:few meta-analyses of mediating processes exist, and those few utilize different strategies. instances are (criticism to correlational data, about having severe missing data problems, and about using inappropriate statistical methods
threats to inferences about external validity in meta-analyses
sampling bias associated with the persons, settings, treatments, outcomes, and times entering a meta-analysis: the persons, settings, treatments, outcomes, or times that are reported in primary studies are rarely sampled randomly
restricted heterogeneity in classes of populations, treatments, outcomes, settings, and times: even if sampling from class were random, inferences about the robustness of a relationship ,ay be wrong if those classes are not themselves heterogeneous
failure to test for heterogeneity in effect sizes: rejection of a homogeneity test implies that systematic variance in effect sizes remains to be accounted for, and further analyses may change results, for example, identifying a variable that moderates the effect.
lack of statistical power for studying disaggregated groups: to explore generalizability, meta-analysts often disaggregate classes of treatments, outcomes, persons, times, or settings to examine whether an effect changes significantly over subgroups.