Please enable JavaScript.

Coggle requires JavaScript to display documents.

Flink– Stream Processing and Batch Processing Platform - Coggle Diagram

- - - - YARN and can obtain data from HDFS and HBaes
      - all formatted input and output of Hadoop
      - the Mappers and Reducers of Hadoop, which can be used together with Flink Operations
- - - - CoGroupedStreams
      - KeyedStream
        
        WindowedStream
      - ConnectedStream
      - AllWindowedStream
      - JoinedStream
    - - indicates the streaming data source, which can be HDFS files , data, or texts
      - processing
        
        Batch processing
        Files / JDBC / HBase / Collections
        
        Stream Processing
        Files / Socket streams / Kafka / RabbitMQ / Flume / Collections / Implement your own
    - - Transformation operators such as map() keyBy() and apply() are used to process data
  - - - RessourceManager of YARN > indicates the ressource management department which schedules and allocates the ressources of the entire cluster in a unified manner
      - TaskManager > performs computing work , an app will be split and assigned multiple TaskManagers for computing
      - JobManager > manages the ressources for apps. JobManager applies to ressourceManager for ressources based on the reqquirments of apps
      - Client > indicates the req initiator which submits app requests and create the data flow
  - - - time windows
        operation based on data window
    - - tumbling windows (whose times dont overlap)
        sliding windows (whose times overlap)
        custom windows
        Session Windows which are considered completed if there is no data within the present time period
  - - - the checkpoint mechanism is the reliability pillar of Flink
        when an exception occurs on an operator in the Flink cluster (unexpected exit) > >the chekpoint mechanism can restore all statuses at a previous time so that all statuses are consistent
      - This Mechanism ensures that when a running app fails < all statuses of the app can be restored from a checkpoint so that data is processed only once.
        Alternatively we can choose to process data at least once.