Please enable JavaScript.

Coggle requires JavaScript to display documents.

Quantitative Methods: Reading #11: Introduction to big data techniques -…

- - - - Increasing functionality to handle massive amounts of data that may come from many sources and exist in a variety of forms (alternative data)
    - - Tools and techniques such as artificial intelligence for analyzing very large or non-linear data sets.
  - - - Refers to all the potentially useful information that is generrated in the economy
    - - Traditional sources: financial markets (trade prices and volumes), company data (annual reports, regulatory filings, sales and earnings figures, and conference calls with analysts), & government economic statistics.
      - Alternative data:
        
        Individuals: social media posts, online reviews, email, and website visits.
        
        Corporate exhaust: bank records and retail scanner data
        
        Internet of Things: Broad network of sensors, chips embedded in numerous smart devicces.
    - - Volume: The amount of data collected in files, records, and tables is very large, representing many mmillions, or even billions, of data points
      - Velocity: The speed and frequency with which tha data are recorded and trnsmitted has accelerated: Periodic -> near-real-time -> Real-time (low latency)
      - Variety: The data are collected from many different sources and in a variety of formats, including structured data (e.g., SQL tables), semistructured data (e.g., HTML code), and unstrutured data (e.g., video messages)
      - Veracity: When Big Data is used for inference or prediction, a "fourth V" comes into play - Veracity - relates to the credibility & reliability of different data sources.
    - - Data science: concerns how we extract informatin from Big Data, describes methods for processing and visualizing data.
      - Processing methods
        
        1. Capture: collecting data and transforming it into usable forms
        
        2. Curation: assuring data quality by adjusting for bad or missing data
        
        3. Storage: archiving and accessing data
        
        4. Search: examining stored data to find needed information
        
        5. Transfer: moving data from their source or s storage medium to there they are needed
      - Visualization techiniques
        
        Structured data: familiar chart, tables and graphs
        
        Less structured data: other methods (world clouds, mid maps, interactive 3D graphics, heat maps, tree diagram,...)
      - Challenges of using BId Data in investment analysis
        
        Quality: dataset have selection biias, missing data, or data outliers
        
        Volume: not sufficient
        
        Appropriateness: not suited for the type of analysis
        
        Dataset with Unstructured data can be extremely difficult to be sourced, cleansed, and organized befor analysis
        
        Challenges Processing
        
        Artificial intelligence
        
        Machine learning
- - - - Attempted to simulate the knowledge base and analytical abilities of human experts in specific problem-solving contexts ("if-then" rules)
    - - Programming based on how our brain learns and processes information (to detect abnormal charges or claims in credit card fraud detection systems)
  - - - involves computer algorithm that extracts knowledge form larfe amounts of data without making any assumptions on the data's underlying probability distribution
      - Algorithm is designed to learn, without human assistance, how to model the output data based on the input data, or to learn how to detect and recognize patterns in the input data.
- - - - When the machine learns the training dataset too precisely, treats noise as true parameters -> the model is too complex to describe a different dataset.
        => Identify spurious patterns and "false" relationships -> prediction errors and incorrect output forecasts
    - - Underfitting occurs when the ML model treats true parameters as noise -> the model is too simplistic and cannot recognize relationships within the training dataseet.
        => Fail to fully discover patterns that underlie the data
    - - ML models are not explicitly programmed -> outcomes are produced based on ML models that are not entirely explainable or understood.
  - - - Computers learn to model relationships based on labeled training data. In supervised learning, inputs and outputs are labeled, or indentified, for the algorithm. After being trained for the labeled data, the trained algorithms are used to model or predict outcomes for new datasets.
        ex: forecast future returns on a stock or predict local stock market performance
    - - Computers are not given labeled data but instead are given only unlabeled data form which the algorithm seeks to describe the data and their structure.
        ex: group companies into peer groups based on their characteristics rather than using standard sector or country groupings
    - - Computers use neural works, often with many hidden layers, to perform multistage, non-linear data processing to identify, can use *supervised or unsupervised machine learning approachers to develop an understanding of simple concepts that informs analysis of more complex concepts, discover insights and identify relationships that were previously too difficult or too time consuming to uncover
        ex: image, pattern, and speech recognition
  - - - Definition
        
        Analysis of unstructured data in text or voice forms
        
        Ex: Analyzing the frequency of words and phrases.
      - Application
        
        Evaluate company regulatory filings, written reports, quarterly earning calls, social media, email, internet postings, surveys.
    - - Definition
        
        Use of computer science, AI and linguistic to develop computer programs to analyze and interpret human language.
        
        Ex: Translation, speech recognition, text mining, sentiment analysis.
      - Application
        
        Review employee voice and communications.
        
        Evalute large volumes of research reports, call transcripts, and other text- and audio-based data.