Please enable JavaScript.

Coggle requires JavaScript to display documents.

Hadoop - Coggle Diagram

- - - - HDFS
        
        What
        
        is the primary data storage system used by Hadoop applications
        
        How
        
        takes in data, it breaks the information down into separate blocks and distributes them to different nodes in a cluster
        
        NameNode
        
        keeps track of where file data is kept in the cluster
        
        knows which DataNode contains which blocks
        
        where the DataNodes reside within the machine cluster
        
        manages access to the files
        
        CRUD(creates, reads, writes, deletes
        
        data block replication across the DataNodes
        
        multiple DataNode
        
        separate blocks and distributed among the various DataNodes for storage
        
        Blocks are also replicated across nodes, for
        highly efficient parallel processing
        
        Why
        
        Cost effectiveness
        
        Fast recovery from hardware failure
        
        Large data set storage
        
        Portability
        
        Streaming data access
    - - YARN
        
        What
        
        Yet Another Resource Negotiator) is a resource management layer in Hadoop
        
        How
        
        Client
        
        It submits map-reduce jobs
        
        Resource Manager
        
        Scheduler
        
        It performs scheduling based on the allocated application and available resources
        
        Application manager
        
        accepting the application and negotiating the first container from the resource manager
        
        restarts the Application Manager container if a task fails
        
        It is the master daemon of YARN is responsible for resource assignment and management among all the applications
        
        Node Manager
        
        It monitors resource usage
        
        log management
        
        creating the container process
        
        start it on the request of Application master
        
        Application Master
        
        4 more items...
        
        Why
        
        hadoop 1;x batch processing, scalability issues
        
        MapReduce for processing big datasets.
        
        YARN clusters are now able to run stream data processing
        
        interactive querying side by side with MapReduce batch jobs
        
        overcoming the shortcomings of Hadoop 1.x
    - - MapReduce
        
        What
        
        is a programming model or pattern within the Hadoop framework that is used to access big data stored in the HDFS
        
        How
        
        Map
        
        is first split into smaller blocks. Each block is then assigned to a mapper for processing.
        
        all the mappers complete processing
        
        shufflling an sorting
        
        Reducer
        
        2 more items...
        
        Why
        
        Cost effectiveness
        
        Fast recovery from hardware failure
        
        Large data set storage
        
        Portability
        
        Streaming data access
- - - - Process the data with MapReduce
        
        Divide the tasks with YARN