Please enable JavaScript.

Coggle requires JavaScript to display documents.

Spark2x - In-memory Distributed Computing Engine - Coggle Diagram

- - - - Hive
      - Mahout
      - Flume
    - - Hadoop
      - Docker
      - Mesos
    - - HBase
      - Elastic
      - Kafka
      - MySql
- - - - Spark Streaming
      - MLib
      - Structured Streaming / new function of Spark 2.0
      - GraphX
      - Spark Sql
      - SparkR
      - Spark Core
      - YARN
      - Standalone
      - Mesos
    - - RDD - Resilient Distributed Datasets
        
        RDD dependencies
        
        Wide
        
        each parent RDD can have multiple RDD childs
        example >> groupByKey
        
        Narrow
        
        each parent RDD have only one child RDD
        example >> filter , map , union
        
        Stage Devision of RDD
        
        RDD Operators
        
        Transformation > invoked to generate new RDD from 1 or more existing RDDs such an operator initiates a job only when an action operator is invoked
        
        map / flatMap , filter and reduceByKey
        
        Action > a job is immediately started when action operators are invoked
        
        take / count / saveAsTextFile
        
        elastic readOnly partitioned distributed datasets
        only sequence of operations are stored from RDDs meaning it dosent matter the order of RDDs
        RDD is stored in the cluster as partitions
        stored in memory and written to disks when the memory is insufficient
    - - Driver > responsible for the application business logic and operation planning (DAG)
      - ApplicationMaster > manages app ressources and applies for ressources from RessourceManager as needed
      - Client > demand side that submits requirments (applications)
      - RessourceManager > ressourceMgn department that centrally schedules and distributes ressources in the entire cluster
      - NodeManager > RessourceManagement of the current node (not perm)
      - Excuter > actual executor of a task . an application for multiiple executors to compute
        An executor of Spark can run multiple tasks at the same time.
    - - Client Operation PRocess
      - Cluster Operation Process
      - Diffrences between YARN-Client and YARN-Cluster
        
        diffrence lie in applicationMaster
        YARN- client is suitable for testing , whereas YARN-Cluster is suitable for production
        if the task submission node in YARN-Client mode is down and the entire task fails << such a situation in YARN-Cluster mode will not affect the entire task
      - In Spark on YARN mode, executors cannot be initiated for tasks on nodes without Node Manager.
    - - JobHistory / JDBCServer...
  - - - module for processing structured data
        in Spark Application , SQL statments or DataFrame APIs can be synchronized used for querying structured data
        
        just a module that can parse SQL language into RDDs and use Spark Core to Execute
    - - DataSets
      - DataFrame
    - - Diffrences
        
        the excution engine of Spark SQl is Spark Core and the default excution engine of Hive is MapReduce
        the excution speed of Spark SQl is 10to100 times faster than Hive
        Spark does not supp buckets and Hive does
      - Dependencies
        
        Spark Sql depends on the metadata of Hive / compatible with most syntax and functions of Hive / can use the custom functions of Hive
  - - - real-engine computing franework built on the Spark which expand the capability for processing massive streaming data
        
        Data can be injected from many sources like Kafka , HDFS and can be processed using complex algorithms expressed with high level function like Map , Reduce , Join and Window
        Finally processed data can be pushed out to file systems and databases