Please enable JavaScript.

Coggle requires JavaScript to display documents.

TOOLS - Coggle Diagram

- - - - Responsible for transferring data from one app to another mostly by asynchronous queue
      - Patterns
        
        Point to Point
        
        Producer send msg in the queue and only one Consumer can retreive it.
        
        When the msg is consumed it disappears from the queue
        
        Exp: Order syst => Order made - put in queue - handle buy one consumer - Disappear
        
        Publish-Subscribe (Pub-Sub)
        
        Msg are persisted in a Topic, sent by Publishers and retreived by Subscriber
        
        Consumers can subscribe to many topics and consume all msgs
        
        Exp: Tv publish channels - anyone subscribe to a set of channels - Watch channel when available
    - - Reliability − Distributed, partitioned, replicated and fault tolerance.
      - Scalability − scales easily without down time by adding node in the cluster ...
      - Durability − "Distributed commit log" (Msg are saved on disk quickly)
      - Performance − Stable when managing TB of msgs
    - - Metrics - Centralise data from distributed app for monitoring
      - Log Aggregation Solution - collect, format and centralise logs from apps for consumers
      - Stream Processing - Ingest a continuous data stream to quickly analyse and transfert in real time
      - Messaging - Manage a huge amount of messages or data
    - - Kafka performance degrades if there is message updating
      - General topic selection is not supported by Kafka, exact topic name is required.
      - Some message paradigms (point-to-point queues, request/reply) are not supported by Kafka
      - Kafka does not have a complete set of monitoring tools.
  - - - Topics - Stream of messages belonging to a category
      - Partition - Part of a topic which handle an arbitrary amount of data
      - Offset - unique sequence id (index) of a partition message
      - Replicas - Backup of a partition to prevent data loss
      - Brokers - Resposible for maintaining the published data
      - Cluster - Kafka's having more than one broker
      - Producers - Publisher of msgs to one or more topics
      - Consumers - Subscriber to topics to pull data from brokers
      - Leader - node (broker) responsible for reading/writing of the given partition
      - Follower - Node which follows leader instructions and have this partition's replicas.
      - :warning:
        
        Leader fails => a follower becomes leader and retrieve data of the preview leader
        
        Topic with N partitions + M brokers => N == M (1 part/brok) | N > M ( 1part on N broks) | N < M (x part/brok No load balance)
      - Replication factor - Number of partition replicas or copies
      - In-Sync Replica (ISR) - Replica up to date with the partition's leader
      - Consumer group - Collection of consumers who work together to ingest data of the same topic
    - - Producers API - API used by Publisher to send stream datas to topics
      - Consumer API - API allows an app to subscribe to topics
      - Streams API - API allows an app to use stream processing architectures to process data
      - Connect API - API connects topic to apps. Exp: capture all DB updates in a topic
    - - What? Distributed configuration and synchronization service.
      - Roles
        
        Maintain and coordinate the kafka cluster => Notify the cluster/consumer in case of new/delete topic/broker
        
        Election of leader => Allows brokers to select the leader of given partition
        
        Save metadata of cluster => save state information of broker (topics, brokers, consumer offsets ...)
      - :warning: After 2.8 it is possible to use Kafka without zookeeper but not before
  - - - 1- Producer send msg to topic
        
        2- Broker stores messages in topic partions
        (Load balance)
        
        3- Consumer subscribes to topic
        
        4- Kafka provides current offset to the new consumer
        and stores the couple in zookeeper
        
        5- Consumer request Kafka in regular interval for new msg
        
        6- Once Kafka receives a producer msg
        it forwards to the consumer
        
        1 more item...
    - - 1- Producer send msg to topic
        
        2- Broker stores messages in topic partions
        (Load balance)
        
        3- Consumer subscribes to topic with Group-ID
        
        4- Kafka provides current offset to the new consumer
        and stores the couple in zookeeper
        
        5- Consumer request Kafka in regular interval for new msg
        
        6- Once Kafka receives a producer msg
        it forwards to the consumer
        
        1 more item...
  - - - 1- Install Java and Download Kafka related to Spring version
      - 2- Extract and edit config files for Kafka and zookeeper
      - 3- Start zooKeeper in CMD : bin/zookeeper-server-start.sh config/zookeeper.properties
      - 4- Start Kafka in another CMD: bin/kafka-server-start.sh config/server.properties
      - 5- Create a Topic in CMD : bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
      - 6- Create a Producer in CMD: bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
      - Create a Consumer in CMD: bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
    - - 1- Create kafka config files of your brokers by setting the brokerId, the port and the log name
      - 3- Start each broker in different CMD with the correspondent config file
      - 2- Start zookeeper in CMD
      - 4- Create topic with replication factor params N for M Brokers : N <= M | N%M == 0
  - - - Differences
        
        Implement the request/Reply, Point-to-Point and Pub-Sub pattern
        
        The message is delete in the queue after consomption
        
        Queue => Topic but before the Queue you have an Exchange which is the entry point of RabbitMq
        
        JMS is a FIFO queue # Kafka ensures that partition are sent in the message order
        
        RabbitMQ has a better monitoring tools
        
        Delivering the message is not guaranteed with RabbitMQ but it is with Kafka
      - RabbitMQ Concepts
        
        Producer/Consumer
        
        Exchange - Receive message from Producer and binds with Queues
        
        Queue - Buffer that store message
        
        Binding - Link between queue and an exchange
        
        Task - Technical data that is sent to be exchanged
        
        Routing key (address) - message attribut that is used by the exchange to route it
        
        Exchange Type
        
        Direct exchange - Routing key == Binding key (One Q)
        
        Topic Exchange - Routing key match with the binding key pattern (path.*.nbf) (Many Q)
        
        Fanout Exchange - Message bind with all queues (Many Q)
    - - 1- Import kafka dependency
      - 2- Create a producer bean with serializer, server host ...
      - 3- Create a Consumer bean with host, groupID, serializer ...
        Add @ EnableKafka on the config File
      - 4- Call the producer send method with Topic Name and Message
      - 5- Use @ KafkaListener (topics= "...", groupId = "...") above a method to get the message
- - - - What? Open-source Distributed VCS and SCM (Source code management)
      - VCS
        
        Version Control System (VCS) = software that helps developers to work together and maintain a complete history of their work :
        
        Types
        
        Centralized Version Control System (CVCS) – Client/Server
        
        Distributed/Decentralized version control System (DVCS) – Peer-to-peer
        
        Localized Version Control System (LVCS) – Standalone
    - - Open source – release under the GPL (General Public Licence) Licence
      - Scalable – Easily handle the users increases
      - Distributed – No central repository, each user has their own repository and can push the changes to a remote repository
      - Security – Git uses the SHA1 (Secure Hash Function) to name and identify objects within its repository
      - Speed – Core part Written in C makes Git very fast for local or remote operations
      - Supports non-linear development – A branch represents a single commit and a full branch structure can be constructed with the parental commit.
      - Branching and Merging – Create, delete and merge on branches
    - - Save time – The speed feature allows us to save time
      - Offline working – Because of distributed feature, we can work locally without internet
      - Undo Mistakes – Undo option for almost everything
      - Track the changes – Track changes with Diff, Log and Status
    - - Git is a distributed VCS – SVN is a Centralized VCS
      - Git is an SCM – SVN is revision control
      - Git has a cloned repository – SVN doesn't have
      - Git is more secure than SVN
  - - - git config [--level] param value
        
        git config --global user.name "Franknbf" – Set a name to show in commit history
        
        git config --global user.email "franknbf@gmail.com" – Set email id for commit histories
        
        git config --global core.editor Vim – Set the default editor in case of file edition
        
        git config --list – show the configuration list with value
      - Levels
        
        -- local – default level and sets configuration for the current git (.git/config)
        
        --global – set user-specific configuration (userHome/.gitconfig)
        
        --system – Entire system configuration or all OS users (etc/gitconfig)
    - - GitBash – Windows command line for git (for UNIX distribution the Bash is the native shell)
      - Git GUI – UI alternative for GitBash
      - Gitk [git log options] – Graphical history viewer tool.
    - - Branch – a version of the repository that diverges from the main working project
      - Checkout – act of switching between different versions
      - Cherry-picking – Apply a commit from one branch into another branch
      - Clone – Make a copy of the target repository
      - Fetch – Get branches and tags from one or more other repositories.
      - Head – Last commit in the current branch
      - Index – Staging area between the working directory and repository
      - Master – Default branch of Git
      - Merge – Process to put a forked history back together
      - Origin – Remote repository ref from a project was initially cloned
      - Pull/Pull Request – Fetch changes on the remote repos and merge in local repos
      - Push – Upload local repository content to a remote repository
      - Rebase – change the base of your branch from one commit to another.
      - Remote – shared repository used to exchange changes
      - Repository – Project folder and git metadata
      - Stashing – Switch branch without committing the current branch
      - Tag – Make a point as a specific point in Git history (Light-weighted and Annotated Tag)
      - Upstream And Downstream – reference of Origin And Local repositories
      - Git Revert – Revert a commit
      - Git Reset – undoing changes (Soft, Mixed, Hard)
      - Git Ignore – Untrack files by Git
      - Git Diff – Runs a diff function on Git data sources (Files, branches, commits ...)
      - Git Cheat Sheet – summary of Git quick references
      - Git Flow – Collection of Git cmds for branching model
      - Git squash – Squash previous commits into one
      - Git Rm – Remove individual files or a collection of files
      - Git Fork – Rough copy of a repository
  - - - Similar as SVN Workflow => All devs work on one Main branch
      - Steps
        
        1- Clone project
        
        2- Make changes and commit
        
        3- Rebase/Pull to origin and correct conflicts
        
        4- Push changes to origin
    - - Make all features development in a branch and merge it on the main
      - Steps
        
        1- Create a new branch
        
        2- Make changes and commit
        
        3- Push to remote and create a Merge request
        
        4- Review and resolve feedback
        
        6- Merge the pull request
    - - Use of features branches and multiple primary branches (main, hotfix, release, dev and feature)
      - Steps
        
        1- A develop branch is created from main
        
        2- A release branch is created from develop
        
        3- Feature branches are created from develop
        
        3- When a feature is complete it is merged into the develop branch
        
        4- When the release branch is done it is merged into develop and main
        
        6- If an issue in main is detected a hotfix branch is created from main
        
        6- Once the hotfix is complete it is merged to both develop and main (with sub-tag version)
        
        5- At the end of the spring dev is merged into main with tag version
    - - Make a copy of an open source project on your own Git server
      - Steps
        
        1- A developer 'forks' an 'official' server-side repository. This creates their own server-side copy.
        
        2- The new server-side copy is cloned to their local system.
        
        3- A Git remote path for the 'official' repository is added to the local clone.
        
        4- A new local feature branch is created.
        
        5- The developer makes changes on the new branch.
        
        6- New commits are created for the changes.
        
        7- The branch gets pushed to the developer's own server-side copy.
        
        8- The developer opens a pull request from the new branch to the 'official' repository.
        
        9- The pull request gets approved for merge and is merged into the original server-side repository
  - - - Git init [nameDirectory] – Initialize a blank repository or a old project create without Git
      - Git add [filename | option] – Add files to index(Staging area)
        -A / . => Add all modified files, new files and deleted files
        add a file twice | git reset <filename> => Remove the file from the index
        --ignore-removal => add all without deleted files
        *.java => Add Files by Wildcard
      - Git commit [option] [value] - Record the changes in the repository
        -a => to add file already existed and not new files
        -m "..." => commit with message directly
        -am "..." => -a + -m
        -amend => modify or merge changes with the last commit
      - Git clone [-b branchName] [GitLink] [folderName]
        [-b branchName] => clone a specific branch
        [folderName] => name of the folder
      - https://www.javatpoint.com/git-ignore
      - Git stash [option] – Put your changes on hold while changing branch
        save "..." => saving stashes with message
        list => check the stored stashes
        apply [stashId] => Apply the last stash or a specific stash
        show [-p] => List the modify files with content (-p)
        pop [stashId] => Apply and remove the last stash or a specific stash
        drop [stashId] => remove the last stash or a specific stash
        clear => Delete all stashes
        branch "branchName" => Create a branch and apply last stash on it