Please enable JavaScript.

Coggle requires JavaScript to display documents.

Flume - Massive Logs Aggregation - Coggle Diagram

- - - - The Source recv events or generates events based on special mechanisms. The source can save events to one channel or multiple channels in batches
        The Sources are Classified into
        
        Types
        
        thrift source
        
        avro source
        
        exec source
        
        htttp source
        
        syslog source
        
        spooling dir source
        
        jms source
        
        kafka source
        
        EventDriven Source
        The External source actively sends data to Flume to drive Flume to accept the data
        
        Event polling Source
        Flume periodically obtains data in an active manner
        
        Channel
        The Source must be Associated with at least 1 Channel
        Flume data flows can be sent to different channels based on the header information
        Multiple channels can be configured to transmit data in the properties.properties configuration file of Flume.
        
        The Channel is located between the source and the sink , ch functions similar the queue > temp saves events . when sink successfully sends events to the next-hop channel or the destination, the events are removed from the current channel
        
        Types
        
        Memory Channel
        high throuPut but presistence and reliability, data may be lost
        
        File Channel
        persistence achieved based on the WriteAhead Log (WAL)
        
        JDBC Channel > Embedded derby database
        supp persistence and high reliability > can replace the file channel that also supp persistence
        
        Sink
        Channels supp transactions and provides weak sequence assurance. Channels can connect any quantity of sources and >> sinks
        
        the sink transmits events to the next hop or destination. After the events are successfullt transmitted, they are reoved from the current channel
        The Sink must be bind to a Specific Channel
        
        Sink Types
- - - - During data transmission, if nextHop Flume is faulty or recv data abnormally > the data is automatically switched over to another path