Please enable JavaScript.

Coggle requires JavaScript to display documents.

DAS-C01 - Practice Exams - Coggle Diagram

- - - - Use PySpark to concatenate the Parquet files instead.
  - - - Use EMR File System (EMRFS) to store the data and enable the EMRFS consistent view feature. Launch two separate EMR clusters in two different Availability Zones. Launch a primary cluster with multiple master nodes and create a secondary read replica cluster in a different Availability Zone. Point the two clusters to the same S3 bucket and hbase.rootdir location.
  - - - You can now create Amazon EMR clusters with custom Amazon Machine Images (AMI) running Amazon Linux. This enables you to preload additional software on your AMI and use AMIs that you customize and control.
  - - - ContainerPendingRatio: This is the ratio of pending containers to allocated containers. (ContainerPendingRatio = ContainerPending / ContainerAllocated) You can use this metric to scale a cluster based on container-allocation behavior for varied loads. This is useful for performance tuning.
      - YarnMemoryAvailablePercentage: The percentage of remaining memory available to YARN (YARNMemoryAvailablePercentage = MemoryAvailableMB / MemoryTotalMB). This value is useful for scaling cluster resources based on YARN memory usage.
  - - - Instance fleet uses Spot Blocks which are uninterrupted spot instances and are available for 1 – 6 hours.
  - - - In the CloudFormation template, define an EMR cluster that uses a custom AMI with encrypted root device volume under the CustomAmild property.
- - - - PySpark es un lenguaje de programación compatible con Apache Spark que puede utilizarse para crear diversas plataformas de datos y gestionar análisis de big data. Es una API de Python desarrollada como parte de la colaboración entre Apache Spark y Python fue creado.
  - - - AWS Glue consists of a central data repository known as the AWS Glue Data Catalog, an ETL engine that automatically generates Python code, and a flexible scheduler that handles dependency resolution, job monitoring, and retries. AWS Glue is serverless, so there’s no infrastructure to set up or manage. Use the AWS Glue console to discover your data, transform it, and make it available for search and querying.
  - - - Run multiple crawler jobs instead of just one.
    - - To populate tables in the AWS Glue Data Catalog, you can use Glue APIs or run a Glue crawler. Glue crawlers can scan different data stores and infer schemas and partition structures to populate the catalog with corresponding table definitions.
    - - certainty=1.0 during processing, it indicates that it’s 100 percent certain that it can create the correct schema
      - If no classifier returns a certainty greater than 0.0, AWS Glue returns the default classification string of UNKNOWN
      - Si no es 1.0 retorna el maximo encontrado
  - - - Modify the job properties and enable job metrics to evaluate the required number of DPUs. Change the maximum capacity parameter value and set it to a higher number.
  - - - Since AWS Glue DataBrew is serverless, you do not need to provision or manage any infrastructure to use it. With AWS Glue DataBrew, you can easily and quickly create and execute data preparation recipes through a visual interface. The service automatically scales up or down based on the size and complexity of your data, so you only pay for the resources you use.
  - - - Soluciones
        
        – Activate useS3ListImplementation
        
        – Filtering with Push Down Predicates
        
        Grouping groupFiles
  - - - RPTA: Create a catalog table using the CreateTable API operation. Write an AWS Glue ETL script to update table partitions.
- - - - Select the dataset and create a daily schedule refresh in the dataset settings.
  - - - OpenSearch dashboard is an open-source visualization tool designed to work with OpenSearch. Amazon OpenSearch Service provides an installation of OpenSearch Dashboards with every OpenSearch Service domain. It is a data aggregation and visualization tool that enables you to explore, visualize, analyze, and discover data in real-time with Amazon OpenSearch.
  - - - Amazon Athena is an interactive query service that makes it easy to analyze data in an S3 bucket using standard SQL. You can visualize the results of your Amazon Athena queries in Amazon QuickSight. Connecting to Athena from QuickSight is a 1-click process. There’s no need to provide endpoints, username, and password. Simply select Athena as your data source, select the database and tables you want to analyze, and start visualizing in QuickSight.
- - - - si se le quiere implementar HSM a un cluster existente de redshift toca hacerlo casi manual y migrarlo a un nuevo cluster
  - - - The nodes all participate in parallel query execution, working on data that is distributed as evenly as possible across the slices.
      - The number of files should be a multiple of the number of slices in your cluster.
      - you need to ensure that the files are roughly the same size, between 1 MB and 1 GB after compression.
      - In this scenario, we want to take advantage of Redshift’s massively parallel processing (MPP) architecture and use a more cost-effective and Apache Parquet which is an operationally-efficient file format. Since the Amazon Redshift cluster has 6 nodes with 4 node slices each, its total number of slices is 24. Remember that in loading S3 data to Redshift, the number of files should be a multiple of the number of slices in your cluster.
  - - - DC2 (dense compute) nodes allow you to have compute-intensive data warehouses with local SSD storage included. You choose the number of nodes you need based on data size and performance requirements.
      - DS2 (dense storage) nodes enable you to create large data warehouses using hard disk drives (HDDs). This node type is recommended for substantial data storage needs
  - - - Typically, elastic resize takes 10–15 minutes. AWS recommends using elastic resize when possible.
    - - During the resize operation, the cluster is read-only. Typically, classic resize takes 2 hours–2 days or longer, depending on your data’s size.
        
        Snapshot and restore with classic resize – To keep your cluster available during a classic resize, you can first make a copy of an existing cluster, then resize the new cluster.
  - - - Implement a temporary staging table to hold the data for transformation.
  - - - Amazon Redshift logs information about connections and user activities in your database. These logs help you monitor the database for security and troubleshooting purposes, which is often referred to as database auditing. The records are stored in Amazon S3 buckets. These provide convenient access with data security features for users responsible for monitoring activities in the database.
      - Amazon Redshift logs information in the following log files:
      - Connection log — logs authentication attempts, and connections and disconnections.
      - User log — logs information about changes to database user definitions.
      - User activity log — logs each query before it is run on the database.
      - The log files rely on Amazon S3 permissions rather than database permissions to perform queries against the tables.
  - - - Clave : is more suitable for frequently changing data
    - - Se organizan por KEY y las tablas quedan guardatas fisicamente juntas, la veo para JOIN
      - Clave: Esta es la busqueda que se espera en la pregunta
    - - ALL distribution is appropriate only for relatively slow-moving tables; that is, tables that are not updated frequently or extensively. Because the cost of redistributing small tables during a query is low, there isn’t a significant benefit to define small dimension tables as DISTSTYLE ALL.
      - Clave: is not updated frequently
- - - - Use random partition keys and adjust accordingly to distribute the hash key space evenly across shards.
        
        It is important to spend some time considering the expected data flow rate for your stream to find the appropriate capacity. Choosing a good partition key strategy helps you take full advantage of the provisioned capacity and avoid hot shards. Monitoring your stream metrics and setting alarm thresholds helps you gain the visibility you need to make better scaling decisions. To update the shard count, Kinesis Data Streams performs splits or merges on individual shards. It can cause short-lived shards to be created, in addition to the final shards.
      - Use the UpdateShardCount API in Amazon Kinesis to increase the number of shards in the data stream.
        
        The UpdateShardCount action is useful when you need to scale your stream, up or down, to a specific number of shards, and you provide that number as a parameter in the API call. The command executes a series of SplitShard and MergeShards actions as needed to reach the explicit number of shards that you specified.
  - - - PutRecordBatch API is only available in Kinesis Data Firehose and not in Kinesis Data Stream.
- - - - per-query limit
      - per-workgroup limit (a.k.a workgroup-wide data usage control limit)
  - - - Amazon Athena allows you to set two types of cost controls:
        
        – per-query limit
        
        – per-workgroup limit (a.k.a workgroup-wide data usage control limit)
  - - - Set data limits in the per query data usage control.
- - - - Architecture: https://media.tutorialsdojo.com/public/s3-glue-data-lake-1_28OCT2023.gif
      - Cuando sumar EMR
        
        AWS Glue can do the job of both determining data schema and processing data transformations. Amazon EMR is more suitable for doing one-time data transformation against terabytes of data since it is designed for cost-effective big data processing.
    - - AWS Lake Formation is integrated with AWS Glue which you can use to create a data catalog that describes available datasets and their appropriate business applications. Lake Formation lets you define policies and control data access with simple “grant and revoke permissions to data” sets at granular levels. You can assign permissions to IAM users, roles, groups, and Active Directory users using federation. You specify permissions on catalog objects (like tables and columns) rather than on buckets and objects.
      - AWS Lake Formation also provides blueprints that you can run for loading and cataloging data. A blueprint allows you to import data from MySQL, Postgres, SQL Server, MariaDB, and Oracle databases running in Amazon RDS or hosted in Amazon EC2. With Lake Formation, you can import data from MySQL, Postgres, SQL Server, MariaDB, and Oracle databases running in Amazon RDS or hosted in Amazon EC2. You can also import from on-premises databases by connecting with Java Database Connectivity (JDBC).
- - - - Optimize the queries by retrieving only from specific partitions.
      - Refactor the Lambda function code by uploading its response payload as an object to an S3 bucket. Generate an S3 presigned URL for the response object and include it in the Lambda function’s response payload.
- - - - stagger window analysis is more suited for analyzing data that comes in at irregular intervals.
    - - It is often used for operations like summing or averaging data over a specified time interval.
    - - A moving average is determined by taking the average of the current and preceding values over a given time period. In the scenario, we want the application to compute moving averages using a sliding 5-minute window. That is, for each new data point sent to the stream, the application calculates the average price using data from the previous 5-minute timeframe.
        
        For instance, this would be appropriate if the requirement was to calculate the average price of a stock every 10 minutes over the last hour. You don’t need a time-based query to compute price changes in real time.
    - - Continuous queries are written in standard SQL and run on the incoming data in real-time. For example, say you’re interested in stock price changes greater than 15 percent. You can use the following query in your Kinesis Data analytics application code.
        https://media.tutorialsdojo.com/Kinesis-data-analytics-continuous-query-sample-transformed.jpeg
    - - this is primarily used when you need to correlate data from different stream sources
  - - - Run the UpdateApplications API every minute to refresh the reference source.
- - - - Este error quiere decir que hay too many shards