Please enable JavaScript.

Coggle requires JavaScript to display documents.

Big Data Concepts and Tools (What Big Data is and how it is changing the…

- - - - Hadoop
        
        -Designed to process big data with reasonable cost and time
        -Hadoop is an open source framework for storing and analyzing massive amounts of distributed, semi and unstructured data
        -open source many people continuously improve it
        -Hadoop clusters run on inexpensive commodity
        hardware so projects can scale-out inexpensively
        -MapReduce + Hadoop = Big Data core technology
        
        how does it work?
        
        -Consists of 2 components: Hadoop Distributed File System (HDFS) and Map Reduce
        -Distributed with some centralization: Data is broken up into “parts,” which are then loaded into a file system (cluster) made up of multiple nodes (machines)
        -Each “part” is replicated multiple times and loaded into the file system for replication and failsafe processing
        -Jobs are distributed to the clients, and once completed, the results are collected and aggregated using MapReduce
        
        Hadoop Cluster
        
        -2 Nodes :
        -Master : Name Node and Job Tracker
        -Slave : Data Node and Task Tracker
        -Data Nodes referred to as storage node where the data is stored
        -Name node keeps track of the files and directories and provides information on where in the cluster data is stored and whether any nodes have failed
        -Job and Task tracker are for processing data and are known as compute node
        -Job Tracker initiates and co-ordinates jobs or the processing of data and dispatches compute tasks to the Task Tracker
      - [MapReduce]
        
        -Goal - achieving high performance with “simple” computers
        -Developed and popularized by Google
        -Good at processing and analyzing large volumes of multi-structured data in a timely manner
        -Distributes the processing of very large multi-structured data files across a large cluster of ordinary machines/processors
        -Used in indexing the Web for search, graph analysis, text analysis, machine learning, …
      - NoSQL (Not only SQL)
        
        -A new style of database which process large volumes of multi-structured data
        -Often works in conjunction with Hadoop
        -Serves discrete data stored among large volumes of multi-structured data to end-users and Big Data applications
        -Examples : Cassandra, MongoDB, CouchDB, Hbase, etc eBay’s Multi Data-Center Deployment
      - HIVE
        
        -Hadoop-based data warehousing-like framework
        developed by Facebook
        -Allows users to write queries in an SQL-like language called HiveQL, which are then converted to MapReduce
      - PIG
        
        Hadoop-based query language developed by Yahoo! It is
        relatively easy to learn and is adept at very deep, very long data pipelines (a limitation of SQL).
- - - - speed, Data is being generated fast and needs to
        be processed fast
        E-Promotions: Based on your current location, your purchase history, what you like
         send promotions right now for store next to you
    - - -Structured Data eg Relational Data (Tables/Transaction/Legacy Data), spreadsheet
        data
        -Semi-structured data eg email, logs, documents
        -Unstructured data eg videos, images, audio files,
        streaming data, graphs, GPS location data, simulation data , streaming data ,etc
        -A single application can be generating/collecting
        many types of data
        -Big Public Data (online, weather, finance, etc)
    - - Data volume increase a lot
    - - The other Vs that define Big Data
        -Veracity : accuracy, quality, truthfulness, trustworthiness
        -Variability : data flows can be inconsistent with periodic peaks
        -Value : provides business value
  - - - The ability to capture, store, and process the huge volume
        of data in a timely manner
    - - The ability to combine data quickly and at reasonable cost
    - - The ability to process the data quickly, as it is captured (i.e.,
        stream analytics)
    - - Security, privacy, ownership, quality issues