Please enable JavaScript.

Coggle requires JavaScript to display documents.

Data Science (Main Skills (Computing infrastructure (Know how to allocate…

- - - - Centroid-based alg
        
        K-means
      - Connectivity based alg
      - Density based alg
      - Probabilistic alg
      - Neural Networks
        
        Performing classification in unsupervised way
      - Autoencoders
- - - - Do you spend the vast majority of your time on projects that take longer than a day?
      - Does data infrastructure have dedicated engineers working on it?
      - Do people in the organization have ways to access basic data without asking a data scientist?
      - Because data science is an interdisciplinary field and data scientists have at least basic skills in many adjacent domains (like engineering, dev ops, product management, math, research, writing, business, etc.) one of the easiest failure modes as a team is if they can’t focus on work that requires that entire set of skills to accomplish. Spending most of your time on ad-hoc requests, supporting simple data access, or doing data pipeline management displaces data science work. Because they can do that work well, it takes a disciplined organization to make sure they don’t have to.
    - - Can you access data without impacting production system performance?
      - Do you spend more time doing analysis than waiting for data?
      - Is there documentation for major schemas?
      - Is instrumentation considered part of a minimum launch-able product?
      - Do you have a process for detecting and fixing bugs in data collection?
    - - Is past research work documented and available in a central location?
      - Does the team have a regular process for reviewing work before sharing it?
      - Do you run experiments to understand the impact of decisions?
      - Can you report negative results without major political pressure?
      - Can the CEO (or other leader) name at least one way the team contributed that quarter?
      - Are data scientists consulted in product and business planning processes?
      - controlled experimentation is the most critical tool in data science’s arsenal
  - - - Reproducibility
        
        No opportunity for code forks
        
        The entirety of the work, from the query of our core ETL tables, to the transforms, visualizations, and write-up, is contained in one file, the Jupyter notebook, RMarkdown, or markdown file.
      - Quality
        
        Review for correctness and precision
        
        By leaning on GitHub’s functionality of pull requests prior to publishing, peer review and version control is put directly into the flow of work.
      - Concumability
        
        Results should be understandable to readers besides the author. Aesthetics should be consistent.
        
        The markdown served by our web-app hides code and uses our internal branded aesthetics, making the work more accessible to less technical readers. The peer review process also provides feedback on writing and communication, which improves the impact of the work.
      - Discoverability
        
        Anyone should be able to find, navigate, and stay up to date on the existing set of work on a topic
        
        The structured meta-data allows for easier navigation through past research. Each post has a set of tags, providing a many-to-one topic inheritance that goes beyond the folder location of the file. Users can subscribe to topics and get notified of a new contribution. Posts can be bookmarked, browsed by author, or perused as a blog feed.
      - Learning
        
        Other researchers should be able to expand their abilities with tools and techniques of others' work
        
        By having previous work easily accessible, it becomes easier to learn from each other. For example, one can review each other’s queries, see the code used for a creative visualization, and discover new data sources. This exposes Data Scientists to new methodologies and coding techniques, speeds up on-boarding, and makes it possible for people outside our team to learn about our field.
- - - - Zipf's Law
        
        Approximates word frequencies, population ranks of cities and countries, etc.
        
        Describes relationship between item frequency and item rank
      - Paretto distribution
        
        Describes HDD error rates, human settlement sizes, moteorite sizes, oil reserve discoveries, distribution of wealth
        
        It can be approximated that 80% of the effects of some particular phenomena are a result of 20% of the phenomena causes
      - Lotka's law
        
        Describes author publication frequency in a given academic field
        
        Is academic discipline-specific
- - - - Classification
        
        Used to predict data with distinct finite classes
        
        Decision Trees
        
        SVM
        
        Neural Networks
        
        Nearest Neighbors
        
        One of the main drivers of DM
      - Regression
        
        Used for predicting continuous numeric output
        
        Linear Regression
        
        Non-linear Regression
    - - Doesn't include pre-labeled data
      - Concept of maximizing intraclass similarity and minimizing the similarity between different classes
      - centroid-style c. (k-means)
        
        hierarchical c.
        
        fuzzy c.
        
        density c.
    - - Essense of DM: taking a set of data and applying statisticsl methods to find interesting and previously unknown patterns
      - Learn patterns of subsets which emerge within a dataset and across instances, which ones emerge frequently, which items are associsted, and which items correlate with others
    - - Can use above methods for its own goal (an end, not a mean)
      - Can also be approached as an excercise in Descriptive Statistics
      - Outliers - data instances that don't seem to readily fit the behavior of the remaining data or a resulting model
      - Fraud detection