Please enable JavaScript.

Coggle requires JavaScript to display documents.

2023-10-19-New Technologies - Coggle Diagram

- - - - Time stamped data
      - Time series data
        
        measurements or events that are
        
        over time
        
        monitored
        
        downsampled
        
        tracked
        
        aggregated
    - - Time series databases efficiently manage time-stamped data
      - They solve challenges like high data ingestion rates, efficient storage, fast time-based queries, and data aggregation
      - TSDB architecture includes ingestion, storage, and query processing layers, with retention policies for data management
      - These databases are specialized for real-time analytics and monitoring application, making them essential for handling time series data
    - - IoT Data Monitoring
      - Financial Market Analysis
      - Energy Management
      - Healthcare Monitoring
      - Network Performance Monitoring
      - Weather Forecasting
      - Log and Event Analysis
      - Monufacturing and Industrial IoT
      - E-commerce and User Behavior Analysis
      - Social Media Analytics
    - - Amazon Timestream
        
        For a US West config:
        
        $0.50/1M write/1 KB size
        
        $0.01/GB scanned (queries)
        
        $0.036/GB stored/hr in memory ($0.03 for magnetic store)
      - Timescale
        
        Prices low as $30/mo for 0.5 CPU 2 GB RAM config
        
        Storage at $0.177/GB/mo
    - - Pros
        
        Strengths/Pros
        
        Efficient Time-Based Queries: Rapid data retrieval for real-time analytics.
        
        Data Compression: Reduced storage costs through efficient data compression.
        
        Built-in Aggregation: Simplified data summarization and visualization.
      - Cons
        
        Weaknesses/Cons
        
        Limited Data Types: Primarily suited for numeric time series data.
        
        Complex Setup: Initial configuration and usage can be complex.
        
        Scalability Challenges: May struggle with extremely large datasets.
    - - -What is downsampling in the context of a Time Series DB?
        
        Downsampling reduces granularity and is done by applying some form of aggregation function like average, sum, min, max, etc., over fixed intervals of time. This operation helps optimize storage space and speeds up query performance, which can be particularly beneficial in settings where quick insights are needed from historical data.
      - -What are the pros and cons of using a Time Series DB instead of a (Real Time/Key Value/Document/Graph) DB?
      - --...for [specific case such as web service monitoring / realtime forecasting etc.]?
      - -What are the components that make up a Time Series DB?
        
        Data Ingestion Layer
        
        Responsible for collecting and importing time-series data into the database from various sources.
        
        Storage Engine
        
        Manages the storage, retrieval, and persistence of time-series data on disk or in-memory.
        
        Indexing Mechanisms
        
        Enable quick search and access to specific data points within the large dataset.
        
        Query Engine
        
        Processes and executes queries from users, converting them into data retrieval or modification operations against the database.
        
        Visualization or Dashboarding Interface
        
        Provides tools for visualizing, monitoring, and analyzing time-series data, often in real-time.
  - - - This makes it easier to perform complex queries and traversals that involve multiple levels of relationships, which can be computationally expensive in relational databases.
  - - - Storing data
      - Access via different terminals (phone/laptop/etc)
      - Real-time synchronization (key feature)
        
        Stores, notifies of and updates changes
        
        Changes received in real-time often using WebSockets
        
        Communication Protocol: WebSockets is a protocol designed for two-way, real-time communication.
        
        Client-Server Interaction: Enables interactive data exchange between a client and a server.
        
        Single, Long-Lived Connection: Unlike HTTP, it maintains a persistent connection for the duration of the interaction.
        
        Reduced Latency: The persistent connection minimizes latency by eliminating the need to establish a new connection for each request.
        
        Real-Time Updates: Ideal for applications requiring real-time data exchange, such as chat apps, online gaming, and financial trading platforms.
    - - Deal with conflicting changes using different strategies:
        
        Last write wins
        
        Manual or custom
    - - Data consistency
      - Online gaming
      - Maybe Git and other collaboration platforms like replit etc
      - Messaging Apps
      - Live updates (twitter, live feeds
    - - For Firebase, one of the most popular: Storage is billed at $5 for each GB/month, evaluated daily. Billing is not affected by the location of your database.
      - Data storage costs
      - Scaling (more servers etc)
      - Backup data costs
    - - Mongo DB
      - -Amazon dynamo DB
      - -Document DB
      - -key-value DB
      - -graph-DB
    - - Realtime collaboration apps, users can access and update data simultaneously
      - -Multiplayer online games, used to sycronize game state and player actions.
      - -Live stream: used for livestream platforms
      - -monitoring network systems, and security systems.
    - - Real-Time Data Updates
      - Low Latency
      - Offline Support
      - Real-Time Notifications
      - Scalability
      - NoSQL Flexibility
      - Collaboration
      - Serverless Options
      - Developer Friendly
      - Security
    - - No SQL Support
      - Costs can add up
      - Time consuming and complex to develop
    - - Explain the difference between a Realtime Database and a traditional relational database. What are the key advantages of using a Realtime Database in specific scenarios?
      - What are the security considerations when using Realtime Databases, and how can you ensure the confidentiality and integrity of the data in a real-time environment?
      - Can you explain how real-time data synchronization impacts application scalability and the challenges that may arise when dealing with a large number of connected clients in a Realtime Database system?