Please enable JavaScript.
Coggle requires JavaScript to display documents.
Open Science - Y2 - Coggle Diagram
Open Science - Y2
The reproducibility crisis
Published research increases knowledge, motivates future research, inspires new products and informs government policy
We need to be confident of the findings of published research and the findings need to be reproducible - if we only ever publish studies with significant findings, the full picture is not seen; this can lead to repetition of null studies and wasting funding
-> It is better to publish all research so people understand the full effects of what is being studied
Results driven-culture puts quality in competition with success (Nosek, Spies & Motyl, 2012)
-> What's best for research - high quality research, published regardless of outcome
-> What's best for researchers - producing a lot of great results
Design limitations
Small samples, low power -
Button et al, 2013 - estimated the median statistical power of studies in the neurosciences is between 8-31%
If 100 studies were conducted with 30% power, all where the effects are true, these will be identified in only 30 studies
-> Reduces the chance of detecting a true effect
-> Reduces the likelihood that a statistically significant result reflects a true effect
Publication bias -
Null findings are harder to publish - published literature is not representative of the research
If we fail to report studies that do not show an effect, we can build up an impression that a study manipulation is highly effective when it actually is ineffective - examples of manipulation that prevent reproduction
-> Failing to report all dependent measures
-> Collecting more data after seeing whether results were significant
-> Failing to report all conditions
-> Stopping data collection after achieving the desired results
-> Rounding down p values (P-hacking)
-> Selectively reporting studies that worked
-> Excluding data after booking at the impact of doing so
-> Claiming to have predicted an unexpected finding
-> Falsely claiming results are unaffected by demographics
-> Falsifying data
de Vries et al, 2018:
Pharmaceutical companies must pre register all trials they intend to use to obtain FDA approval, hence trials with non-significant results, even if unpublished, are accessible
-> Study publication (around half supported) -> outcome reporting bias -> spin -> citation bias (mostly only cite supported hypotheses)
-> Means that all null studies eventually get lost in the bias process and due to spin, the results are manipulated
Evidence-based medicine is the cornerstone of clinical practice, but it is dependent on the quality of evidence upon which it is based - unfortunately, up to half of all randomised controlled trials have never been published
Trials with statistically significant findings are more likely to be published than those without (Dwan et al, 2013)
Negative trials face additional hurdles beyond study publication bias that can result in the disappearance of non-significant results (Boutron et al, 2010; Dwan et al, 2013; Duyx et al, 2017)
Study publication bias involves nonpublication of an entire study, outcome reporting bias refers to non-publication of negative outcomes within a published article or switching the status of ‘non-significant’ primary and ‘significant’ secondary outcomes (Dwan et al, 2013) and spin in the abstract
Positive trials were cited three times as frequently as negative trials
Questionable research practices
P-hacking (this is where you choose the method of analysis works best after the results are collected, with the one that looks best being the one you report), selective reporting, HARKing (Hypothesising after research conducted)
-> P-hacking - transparency is key - repeating analyses with different controls but make sure everything is reported
Fanelli (2010) - 91.5% of psychology / psychiatry papers claim evidence for an A PRIORI hypothesis, compared to 70.2% for space science
Why?
-> Do hypotheses in different disciplines have different truth values (based on prior knowledge, maturity of the discipline)
-> Discipline differences in regor - experimenter effects, non-publication of null findings, questionable research practices
How questionable research practices and publication bias break the research cycle - Munafo et al, 2017:
The cycle = Publish / conduct next experiment -> generate and specify hypothesis -> design study -> conduct study and collect data -> analyse data and test hypotheses -> interpret results ->
The added issue = Publication bias -> failure to control for bias -> Low statistical power -> Poor quality control -> P-hacking -> P-hacking -> publication bias
-> HARKing
Open Research practices - incentives and barriers
Open science is an asset - increasing understanding, access, credit and reproducibility
-In response to the reproducibility crisis, research is reforming
Up to now -
-> Dominance of closed research practices, lack of sharing, lack of methodological detail, publications as static adverts
-> Small research samples, lab-based working structures / small empires
-> Emphasis on novelty and originality of results (story-telling)
Now / future -
-> Shift toward standardised open research; archiving of data/evidence, code, materials; open access; preprints
-> Shift toward multi-site consortia to tackle big questions (cooperative research)
-> Increasing emphasis on transparency and reproducibility; including study pre-registration, registered reports and replication
Open science is a collection of actions designed to make scientific processes more transparent and results more accessible - its goal is to build a more replicable and robust science; it does so using new technologies, altering incentives, and changing attitudes (Spellman et al, 2017)
Requires a paradigm shift, to conflict with the competitive system of science that ranks researchers according to metrics such as the number of their citations - it is about encouraging researchers to collaborate, to share with each other and the public, and to be transparent for the greater good of society (Shearer, 2020)
UNESCO (2021) - more open, transparent, collaborative and inclusive scientific practice, coupled with more accessible and verifiable scientific knowledge subject to scrutiny and critique is a more efficient enterprise that improves the quality, reproducibility and impact of science and thereby the reliability of the evidence needed for robust decision making and policy and increased trust in science
Recommendations for researchers (Button et al, 2013):
Perform a priori power calculation for sample size
Disclose methods and findings transparently
Preregistration of protocol and analysis
Work collaboratively to increase power and replicate findings
Open science framework -
Pre-registration - created before data collected / accessible and a time-stamped record of study design, methods and analysis decisions (can change the pre-registration)
-> UKRN primers - two page summaries
Registered reports - in principle acceptance of paper based on introductions, method, research questions, analysis plan; reduce questionable research practices, prevent biases such as selective reporting and registered reports (alleviate publication bias)
Make study materials and data available -
Increase transparency
Allow replication
Increase citable materials
Sharing data and code is common in quantitative research, but is applicable to any
Open data - FAIR:
Finadable - metadata and data should be easy to find for humans and computers; machine-readable metadata are essential for automatic discovery of datasets and services
Accessible - once the user finds the required dataset, they must be able to actually access it, possibly including authentication and authorisation
Interoperable - needs to be integrated with other data and interoperate with as wide as possible a variety of applications or workflows for analysis, storage and processing
Reusable - to be able to reuse data, metadata and data should be well-described so that they can be replicated and / or combined in different settings
Open data -
Sharing data is a requirement of many funding bodies and scientific journals, and thus sharing data helps to maximise the impact of research investment through wider use, and encourages new avenues of research
Allows replication and increases citations
Information sheets and consent forms contribute to this - consenting to anonymised accessible data
Open research incentives -
Rigorous research with reproducible findings - wider evaluation and scrutiny by scientific community
Increased citations - 97% more downloads in Wiley compared to previous year when data was open access
Increased impact and visibility of research
Public benefit - maximise the impact of research investment
Increase ability of editors, reviewers and researchers to understand research
Increase opportunities for collaboration and new avenues of research
Data and materials have a Digital Object Identifier, which is citable and gives credit to work beyond the research process
Transparency and re-use
Future you will understand what you did, how you did it and why
Research culture reform
Open research takes more time and resources - importance of slow research needs stronger recognition
If you participate and train in Open research methods, you help change the culture
Case study for open research - COVID-19 pandemic
-> State of Open Data Report (2021) - urgent need to understand the virus brought unprecedented collective and collaboration
--> Preprints have been critical in providing latest research, global collaboration increased, and open data accelerated identification of virus spread, infection patterns, incubation periods, symptoms, age groups at risk etc
-> What has the COVID-19 pandemic taught us -
--> Statistics with kind permission from Ana Persic, UNESCO (2022) -
--> 30% of all scientific populations are open access
--> 50% of climate change related publications are open access
--> 85% of all COVID-19 related publications are open access
Fast research - transparency does not always bring rigour -
The model, credited with introducing the UK lockdown, was considered totally unreliable by leading figures - heavy criticism because codes was not accessible
-> However, with time, the code was figured out and it was found that the model was reliable
Open research - perceived barriers
State of Open Data Report (2023) - Do you think researchers currently get sufficient credit for sharing data?
More than 50% said no in every year between 2019-2023
University of Surrey, Open Research Questionnaire (2020 v 2023):
-> The most frequent responses regarding barriers to the uptake of open research practices were lack of dedicated funding and time
-> Few respondents indicated no barriers to open research practices (2020 only; 5.9%)
-> In 2023, perception of barriers was overall lower; lack of info/training was no longer a substantial barrier, but lack of funding, time and incentives remained high
Open data incentives - SODR (2023):
Which of these circumstances would motivate you most to share your data?
-> Citation of research papers was highest, followed by public benefit, increased impact and visibility, full data citation, co-authoring and financial reward
-> Journal requirements was also high but not higher than the rest of the listed incentives
Open data - cross-country comparison (State of Open Data, 2024):
United states - second largest volume of papers, more of a barrier of informing about practice; less data retraction than China, Data Availability Statements changing to data on request, evolving policy and sustained policy development
UK - clear policy frameworks, data linking practice in published research is above average, Data Availability Statements are less common, highest awareness of FAIR data, UK National Data Strategy underscores importance of open data
Germany - complex data policy landscape, no national framework, high percentage of publication, federal government has set goals to strengthen open science with a Research Data Action Plan
Ethiopia - significant research activity growth, more data sharing, research output has increased, high percentage of open data papers compared to other countries, heavy reliance on external funding, which could influence data practice
China - most publications, fastest growth rate, but below average data linking - subject specific repositories may explain this, regulatory frameworks are evolving to respect privacy and mitigate misuse, and specific challenges in balancing openness with data security in genomics, health and technology
-> Measures for the Management of Scientific Data - 2018; mandated government funded research must make data openly accessible where possible
Japan - biggest funders of research - Japan Society for Promotion of Science encourage data sharing through policy, lowest awareness of FAIR, good improvement of data linking, high percentage of Data Availability Statements with international collaboration
-> New research dataset mandate from Cabinet Office stating new research from April 2025 onwards must make resulting publications and their underlying research datasets openly accessible