Please enable JavaScript.
Coggle requires JavaScript to display documents.
Psychometrics module 10: validity generalization and psychometric meta…
Psychometrics module 10: validity generalization and psychometric meta-analysis
Validity Generalization
the error that was occurring from one organization to the other was due to sampling error.
correcting for only sampling error associated with studies is known as performing the "bare bones" validity generalization (VG) study. such studies provide stronger and more realistic estimates of the average observed validity coefficient across studies than is possible in any single study.
sampling error and various other statistical artifacts account for less than 75% of the variability in observed validity coefficients
, nonartifacts (true) variability likely exists and so moderator analysis should be performed. such moderators might include the type of organization or job, when the study was conducted, the type of criterion used in the validation study, and so forth.
from VG to psychometric meta-analysis
the procedures used for VG could be applied to any estimate of association (or effect size) between key variables.
the first two goals of most meta-analyses is to obtain the most accurate and best possible point of estimate of the population effect size
conducting a meta-analysis
compile a list of relevant published and unpublished studies using a variety of sources and computerized search options
must then read all of the papers obtained and formulate hypothesis about potential moderator variables
we need to develop a coding scheme and rules of inclusion and exclusion for the study.
we need to train those who will be doing the coding, and after coding a few studies, we should check fo inter-coder consistency.
code all studies pulling out the key data to conduct our meta-analysis, as well as essential information regarding potential moderators
we actually analyze the data
we draw appropriate conclusions about the effect of interest and potential moderators
best practices
when conducting psychometric meta-analysis, realize that garbage in will equal garbage out
be sure to clearly define the population to which you are interested in generalizing your results. this is just one of many judgement calls that will need to be made during the meta-analysis process
realize that psychometric meta-analysis is simply another tool to add to you methodological toolbox. it will not be a panacea for a series of ill-conceived and ill-conducted studies
document, in detail, all steps completed during your meta-analysis study