Good Research Practices - Meeting 5
Open Science (Spellman, Gilbert, Corker)
Why it's time for change
Practicing Open Science
Motivations for concern
Objections to Open Science
Questionable Research Practices (QRP's)
HARKing
Undisclosed flexibility --> interesting and publishable results
Failures to replicate
Fraud
Lacking access to full methods
File drawer problem
Lacking access to publications
Reporting/utilizing standard statitics
Combined resources for high-powered replications across labs/populations
Many Labs Project
Registered Replication Reports
Reproducibility Project: Psychology
Reduce selection bias
Make process transparent
Increase fidelity
Replication findings disappointing but process is informative:
Difficulty of direct replications
Difficulty reproducing materials and measures across populations/time
No clear interpretation on 'successful replication'
Possibly resulting from bad actors & bad decision
Several published psychologists caught
Incomplete/limited methods or original authors
Short format of articles
Improper archiving of materials
Non-preserved stimuli (media/technology limitations)
Lacking access to analytic procedures/code
Didn't preserve analysis code on statistics softwares
Difficulty reproducing analyses
Inability to publish successful/failed replications
Robust findings become mischaracterized
Payment required for access to scientific publications
Dissatisfaction with null hypothesis testing
Demographics; diversity in researchers changing
Science-wide problem; occurs across several domains
Technology; faster & more efficient research
Insights from psychology; misaligned incentive structure (biases)
Transparency & openness means information is:
(3) Discoverable
(2) Intelligible
(1) Available
For researchers
For teachers
For authors/reviewers
(1) Save all data & materials (error-checking/reanalysis)
(2) Organized, labeled, annotated data/materials
reusable formats, secure online repositories, data descriptors
codebooks, checklists, or research recording templates
(3) Sharing data (born open data)
concern for 'scooping' but benefits from collaboration & communication
Preregister confirmatory research
Differentiating from exploratory;
Decreases harking/ cherry-picking
Reduce Type 1 error
Integrate info on reproducibility in course programs
Project TIER
21 Word solution; fully disclosed methos
Badges for published papers using open practices
Selectively review papers on open-science standards
Suggestions for reviewers:
authors comply with transparency requirements
recognizing real results are messy
showing results are robust to the analytic choices
demanding exact replications if prior inquiries are insufficient
Join large-scale replication projects
Debate whether there is a replication crisis
Failure to replicate is natural process in science
Encourages novel discovery
Funding & reputation of psych tarnished if done publicly
For trustworthy field --> increase openness & transparency
Divide in preference for action
young researchers; push for change
old researchers; urge cautiousness
Injustice of rules
'Open' solutions hinder science; unnecessary & unrealistic burdens
Slows down scientific discovery
Need more time & funding
Excessive effort for researchers; preregistration, all open data/materials etc.
Produces inequities in evaluation
open practices slow down production and publication of research
new norms create disadvantage to newcomers for success
May still prove ineffective to catch fraudsters
Future of Open Science
Main concern = resembles the wild west (no overarching rules)
Moving forward:
making better bricks
meta-analyses practiced broadly and consistently
understanding aggregation of imperfect results
slowing increases but decrease rate of false discoveries
Progress Towards Openness, Transparency, & Reproducibility in cognitive neuroscience (Gilmore et al.)
History of open science practices in cognitive neuroscience (CN)
Disagreement on acceptable reliability in fMRI
Openness & transparency relate directly to:
Methods reproducibility
Results reproducibility
Inferential reprodicibility
regenerating complex computational workflows
amount of methodological/analytical choices creates barriers:
reporting
analysis
preprocessing
data acquisition
experimental design
reliability of workflows
Standardized processing workflows available
Biomedical Infometrics Research Network (BIRN)
Main barriers:
Sharing full results has limited success in neuroimaging
Lacking consensus on appropriate quantitative measures
Data privacy, formats, technology, time & money constraints
Conceptual & statistical problems threatening neuroimaging inferences
Defining extracted BOLD info from MVPA (multi-variate pattern analysis)
Recent initiatives:
Results reproducibility #
Inferential reproducibility #
Methods reproducibility #
Avoiding manual data manipulation steps
Track versions of all software & data
Public access to all code, outputs & data
Less customized experiments; more standardized analytic environment
Openly shared raw BOLD data; permit reanalysis/meta-analysis
Center for Reproducible Neuroscience
long-term, shareable, containerized, modular, fully reproducible cloud-based tools
Systematic measures of factors influencing test/retest relaibility
Addressing issues on long-term within-subject stability in responses
Conducting & publishing results of confirmatory studies
Preprint services, preregistered reports, proper use of stats (statcheck)
Large-scale imaging databases inferential tool for meta-analyses (neurosynth)
Overfitting; risk exaggerated or false results
Machine learning & big data --> neural information pattern analysis (subtle/complex patterns)
Cross-validation; counter overfitting, but still risky through optimalization analysis
Future
Big data accelerates generalizable, robust discoveries on cognition
Difficulty in reporting reproducible information; some respositories exist though
Estimating the Reproducibility of Psychological Science (Open Science Collaboration)
100 published replications; high-power design & original materials
Introduction
Scientific claims gain credence through replicability
Direct replication establishes reproducibility of a finding using new data
Still possibility of irreproducible findings from systematic error
Reasons:
Differences in studies moderate observed effect size
Original has false-positive
Replication has false-negative
Results; No single sufficient indicator for replication success
Other means to evaluate reproducibility (outside 5 indicators)
Despite precautions, weaker evidence produced in replications
Correlational evidence:
Variation in strength of evidence more predictive of replication success
Insights on reproducibility
Direct replications supports reliability of results
doesn't add to theoretical understanding
Failed replications can be from false-negatives
Biased effect sizes; low-power designs, publication bias
Cause for study differences; publication, selection, reporting biases
Reduced with replications; preregistration & preanalysis & confirmatory tests
Limitations
Findings extended to rest of domain or others unknown
Unknown effect of selection bias
Replication increases certainty of findings and promotes innovative science
The Weirdest People in the World (Henrish, Heine, Norenzayan)
Intro/Background
WEIRD societies: Western, Educated, Industrialized, Rich, Demopgraphic
Findings from narrow databases shouldn't be assumed to be universal
Contrast 1: Industrialized Societies vs. Small-scale Societies
Several domains where data is similar between industrialized & small-scale
Industrialized societies were outliers regarding:
Visual illusions, social motivations, folkbiological cognition & spatial cognition
Without empirical support, cannot generalize industrialized to all populations
Contrast 2: Western vs. Non-western Societies
Westerners ares frequent global outliers on several dimensions
Contrast 3: Contemporary Americans vs. Rest of the World
Sufficient data for cross-population comparisons show Americans are exceptional
Even within the western population (outliers among outliers)
Contrast 4: Typical Contemporary American Subjects vs. Other Americans
Many similarities & differences between groups, in unexpected domains
Appear further removed along some dimensions than predecessors
Conclusion:
Can't generalize from subpopulation
Begin building broader, more grounded understanding of human species
Using Nonnaive Participants can Reduce Effect Sizes (Chandler, Paelacci, Peer, Mueller, Ratliff)
Assumption that participants are naive to experimental materials isn't always true
Investigated affect of prior exposure on subsequent task results
Results:
Nonnaivite reduced observed effect sizes
25 % reduction when performing same tasks on 2 separate occasions
Moderators of the decline of effect sizes
Effect of Variables of Theoretical Interest & higher-order interactions on tasks
Memory for participation in prior studies was high (35-80%)
Reporting prior participation doesn't predict effect size attenuation