Please enable JavaScript.

Coggle requires JavaScript to display documents.

CAREER - INTERVIEW - Hadoop - questions prep (1) (Hadoop HDFS interview…

- - - - RDBMS
        
        structured data
        known schema
      - Hadoop
        
        structured/unstructured/semi
    - - RDBMS
        
        limited/no processing capability
      - Hadoop
        
        data distributed across clusters can be processed in parallel
    - - RDBMS
        
        based on a 'schema on write'
        
        (?)
      - Hadoop
        
        based on a 'schema on read'
        
        (?)
    - - RDBMS
        
        fast reads because known schema
      - Hadoop
        
        HDFS writes are fast because no validation
    - - RDBMS
        
        licensed software, you have to paye
      - Hadoop
        
        open source, no payment
    - - RDBMS
        
        OLTP (Online Transactional Processing)
      - Hadoop
        
        data discovery, analystics, OLAP
  - - - large data
      - complex data sets
      - hard to process with traditional RDBMS
      - difficult to capture, store, search, transfer, analyze and visualize
    - - Volume
        
        Petabytes or Exabytes
      - Velocity
        
        yesterday's data is already old
        
        social media contributes a lot to its velocity
      - Variety
        
        heterogeneous data
        
        csv, audio, video, ...
      - Veracity
        
        inconsistency, incompletedness
        
        volume is often the reason for the lack of accuracy
      - Value
        
        big data that cannot be turned into value is useless
  - - - What?
        
        Storage unit of Hadoop
      - How?
        
        Stores different kinds of data as blocks in a distributed environment.
      - Other characteristics
        
        Master-slave topology
      - Components
        
        NameNode
        
        NameNode is the master node in the distributed environment and it maintains the metadata information for the blocks of data stored in HDFS like block location, replication factors etc.
        
        DataNode
        
        DataNodes are the slave nodes, which are responsible for storing data in the HDFS. NameNode manages all the DataNodes.
    - - What?
        
        Processing framework in Hadoop
      - How?
        
        manages resources and provides an execution environment to the processes
      - Components
        
        ResourceManager
        
        It receives the processing requests, and then passes the parts of requests to corresponding NodeManagers accordingly, where the actual processing takes place. It allocates resources to applications based on the needs.
        
        NodeManager
        
        NodeManager is installed on every DataNode and it is responsible for the execution of the task on every single DataNode.
  - - - Generally approach this question by first explaining the HDFS daemons i.e. NameNode, DataNode and Secondary NameNode, and then moving on to the YARN daemons i.e. ResourceManager and NodeManager, and lastly explaining the JobHistoryServer.
    - - NameNode
        
        It is the master node which is responsible for storing the metadata of all the files and directories. It has information about blocks, that make a file, and where those blocks are located in the cluster.
      - DataNode
        
        It is the slave node that contains the actual data.
      - Secondary NameNode
        
        It periodically merges the changes (edit log) with the FsImage (Filesystem Image), present in the NameNode. It stores the modified FsImage into persistent storage, which can be used in case of failure of NameNode.
      - ResourceManager
        
        It is the central authority that manages resources and schedule applications running on top of YARN.
      - NodeManager
        
        It runs on slave machines, and is responsible for launching the application’s containers (where applications execute their part), monitoring their resource usage (CPU, memory, disk, network) and reporting these to the ResourceManager.
      - JobHistoryServer
        
        It maintains information about MapReduce jobs after the Application Master terminates.
- - - - This is an important question and while answering this question, we have to mainly focus on two points i.e. Passive NameNode and YARN architecture.
    - - Hadoop 1.x
        
        NameNode is a single point of failure
      - Hadoop 2.x
        
        Active & Passive NameNodes
      - If the active “NameNode” fails, the passive “NameNode” takes charge. Because of this, high availability can be achieved in Hadoop 2.x.
    - - Hadoop 1.x
        
        MRV1 (Job Tracker & Task Tracker)
      - Hadoop 2.x
        
        MRV2/YARN (ResourceManager & NodeManager)
      - Also, in Hadoop 2.x, YARN provides a central resource manager. With YARN, you can now run multiple applications in Hadoop, all sharing a common resource. MRV2 is a particular type of distributed application that runs the MapReduce framework on top of YARN. Other tools can also perform data processing via YARN, which was a problem in Hadoop 1.x.