Please enable JavaScript.

Coggle requires JavaScript to display documents.

Reproducible Research (What is RR :!?: (A study is reproducible, if (the…

- - - - code and data
  - - - where likelihood to replicate fully is time sensitive
  - - - investigators
      - data
      - analytical methods
      - laboratories
      - instruments
- - - - results can't be reproduced by independent researcher
    - - results cannot seem to be reproduced by independent researchers
    - - results could be reproduced with extreme efforts by independent researcher
    - - results could be reproduced with considerable efforts by independent researcher
    - - can be easily reproduced with at most 15 minutes, require proprietary source packages e.g. SAS
    - - can be easily reproduced with at most 15 minutes, require only standard freely available tools e.g. R, Python
    - - Reviewable Research
        
        descriptions of research methods can be independently assessed
        
        e.g. traditional peer reviewed
      - Replicable research
        
        tools made available, allow one to duplicate results of research
      - Confirmable Research
        
        main conclusion of research can be attained independently
        w/o use of software provided by author
      - Auditable Research
        
        sufficient records (data & software included) archived,
        to defend research in future,
        in case of differences of independent findings
      - Open or Reproducible Research
        
        Auditable research made openly available
        
        comprised well documented and fully open, publicly avaliable code and data
        
        this allows
        
        a) fully audit of computational procedure
        
        b) replicate & independently reproduce results of research
        
        c) extend results, apply methods to new probles
    - - Good
        
        Use code
        
        minimize pointing and clicking (RStudio). Mention availability of code
      - Better
        
        Use version control
        
        keeps track of changes, fix bugs,
        improve project management
      - Best
        
        Use embedded narrative and code to explicitly link code, text and data
        
        save your time, reviewers time, improve code
        
        e.g. RStudio, Git & Github combo
  - - - share in open repositories
        
        data, software, workflows,
        details of computational environment
    - - persistent links should appear in published doc
      - include permanent identifier for data, code, and digital artifacts
    - - citation, standard practice
    - - adequately document digital scholarly artifacts
    - - should, by Journals
    - - to publish digital scholarly objects
  - - - an analysis can be reproducible and still be wrong
- - - - literate program
        
        weaved
        
        to produce human readable doc
        
        tangled
        
        to produce machine readable doc
      - a useful way to put text, data, code and output all in one doc
        incld. graphical results
        
        needs
        
        a programming language
        
        Knitr supports a variety
        of documentation languages
        
        knitr is a programming language specific tool
        
        a documentation language
      - Pros :+1:
        
        logical order of text & code
        
        code is live
        
        data, results are automatically updated
        
        to reflect external changes
      - cons :-1:
        
        difficult to read, a lot of codes
        
        substantially slow down processing of documents
      - evolution
        
        HTML :arrow_left:
        
        Latex :arrow_left:
        
        Sweave :arrow_left:
        
        restructured text :arrow_left:
        
        Markdown
        
        idea origin: Don Knuth
  - - - Taverna
      - Sumatra
      - Galaxy
    - - tracking of chronology & origin of research objects e.g.
        data, source code, figures & results
  - - - Github
      - Google Code
  - - - that represent sequences of commands & calculations as an interactive worksheet with pretty printing and integrated displays,
        decoupling content (the data, calculations) from representation (PDF, HTML), so that the same research content can be presented in multiple ways
      - Rstudio with knit R, IPython
- - - - reproduction of the original results using the same tools
        
        original author/same machine
        
        someone same lab/same machine
        
        someone in different lab
    - - completely independent reproduction
        
        based on only text description
        
        w/o access to codes
  - - - Reproducibility
        
        minimum standard
        
        for assessing the value of scientific claims
        
        particularly, when full independent replication of a study is not feasible
      - replication
        
        Full replication is scientific gold standard
        
        the confirmation of results & conclusions from one study obtained independently in another
      - sole publication not reproducible
  - - - repeatability & reproducibility
        
        re-run code and get same results
      - repeatability
        
        re-run code and get different results
      - reproducibility
        
        obtain the same results
        using a different program
- - - - keep track of things i.e. version control
      - use software whose operations can be coded
      - don't save output
      - save data in non-proprietary formats
      - Decide to do it at the start
    - - tools for researchers, developers
        
        software tools
      - dataset repositories
        
        servers
        
        website
      - Rights framework for dataset
        
        protocols
        
        standards for exchanging data
      - other necessary information
    - - Begin with the final product in mind
      - use literate programming
      - keep history of changes via code versioning & sharing
      - Get basic statistics right
      - set stringent cutoffs, correct p-values for multiple testing
      - Be critical
        
        consider batch effects
        
        visualize
        
        do sanity checks
        
        use random controls
        
        control validation
      - Follow reporting guidelines
  - - - supports variety documentation language
        
        RMarkdown
        
        LaTeX
        
        HTML
    - - short/medium length technical doc
      - tutorials
      - reports - periodical
      - data pre-processing
    - - long research articles
      - complex time-consuming computations
      - documents that require precise formatting
- - - - open research
        
        scientific knowledge of all kinds should be openly shared, as early as practical in the discovery process
        
        need to reward publication of research output along the process
        
        not just based on article is published
    - - trends
        
        scientific research becoming more computational
        
        Scientific computing will become dramatically transparent
      - the success and credibility of Science should depend on willingness of scientists towards independent testing and replication
      - scientific evidence is strengthened when important findings are replicated by independent parties
  - - - people are interested in the science, high-level and not the experimental setup
    - - pseudo-code, a great way of communicating ideas quickly and clearly, - not understand the subtle implementation details
    - - any limitations of methods/proofs will be obvious to careful reader
    - - others won't understand how the code actually work or doesn't actually work
- - - - share work, and data product that allows others to replicate, and produce the same results
      - team will able to capitalize on its collective knowledge, and advances within the team
      - data science is about collaboration
    - - Establish audit trail & explain to satisfy regulatory
        concerns and public demand
        
        especially if applications and
        decision-making impact users
    - - can determine model performance based on comparison of new analysis with reproduced old analysis
      - can properly justify adding creative sophistication or complexity to analysis
      - can avoid repeating mistakes