Please enable JavaScript.

Coggle requires JavaScript to display documents.

Data Stores (Analytics analytics (Data Pipeline (Data Pipeline (Product…

- - - - Memcached vs Reds
        
        Redis
        
        1 rich set of features --> wide range of use cases
        
        2 encryption
        
        in-transit
        
        at-rest
        
        3 geospatial indexing
        
        4 complex data types
        
        5 automatic failover
        
        6 backup and restore
        
        7 multiple databases
        
        Memcached
        
        2 run large nodes with multiple cores or threads
        
        3 ability to scale out and in
        
        4 cache objects, such as a database
        
        1 designed for simplicity
      - Use Cases
      - Managed Components
      - Function
        deploy, run, and scale popular open source compatible in-memory data stores
      - Use Case
        
        Leaderboards for gaming
        -data constantly changing
        -need for real-time information
        -move from application to elasticache
  - - - DBs Supported
        
        PostgreSQL
        
        MariaDB
        
        Oracle
        
        SQL Server
        
        MySQL
      - Features
        
        Lower administrative burden
        
        Easy to use
        
        CLI
        
        API Calls
        
        Managment Console
        
        Automatic software patching
        
        Performance
        
        General Purpose SSD Storage
        
        3 - 3000 IOPS per provisioned GB
        
        Provisioned IOPS SSD Storage
        
        Up to 40 000 IOPS per database instance
        
        Optimized for transactional OLTP database workloads
        
        Scalability
        
        Scalability
        
        Easy storage scaling
        -As your storage requirements grow, you can also provision additional storage.
        -The Amazon Aurora engine will automatically grow the size of your database volume as your database storage needs grow, up to a maximum of 64 TB or a maximum you define.
        -The MySQL, MariaDB, Oracle, and PostgreSQL engines allow you to scale up to 32 TB of storage and
        -SQL Server supports up to 16 TB. Storage scaling is on-the-fly with zero downtime.
        
        Read Replicas
        -Read Replicas make it easy to elastically scale out beyond the capacity constraints of a single DB instance for read-heavy database workloads.
        -You can create one or more replicas of a given source DB instance and serve high-volume application read traffic from multiple copies of your data, thereby increasing aggregate read throughput.
        -Read replicas are available in Amazon RDS for MySQL, MariaDB, PostgreSQL, and Oracle as well as Amazon Aurora.
        
        Push-button compute scaling
        You can scale the compute and memory resources powering your deployment up or down, up to a maximum of 32 vCPUs and 244 GiB of RAM
        
        Availability and durability
        
        Automated backups
        -The automated backup feature of Amazon RDS enables point-in-time recovery for your database instance.
        -Amazon RDS will backup your database and transaction logs and store both for a user-specified retention period. This allows you to restore your database instance to any second during your retention period, up to the last five minutes.
        -Your automatic backup retention period can be configured to up to thirty-five days.
        
        Database snapshots
        -Database snapshots are user-initiated backups of your instance stored in Amazon S3 that are kept until you explicitly delete them.
        -You can create a new instance from a database snapshots whenever you desire.
        -Although database snapshots serve operationally as full backups, you are billed only for incremental storage use.
        
        Security
        
        Manageability
        
        Cost-effectiveness
      - FAQ
  - - - Nike "Launch"
        
        From cassandraDB/couchbase to DynamoDB
        
        Products
        
        sneakers app
        
        nike.com
        
        requirements
        
        a lot of users
        
        a lot of data source needs
        
        high throughput
        
        devops
        
        you build, it you own it (and operate it)
        
        need to make it simpler
        
        only a big spike with new sneakers launch for 8 min.
    - - Performance at scale
        
        Key-value and document data models
        
        Microsecond latency with DynamoDB Accelerator (DAX)
        --> In-memory
        
        Automated global replication with global tables
        
        Real-time data processing with DynamoDB Streams
      - Serverless
        
        Read/write capacity modes
        
        On-demand
        
        accommodates your workloads as they ramp up or down to any previously reached traffic level
        
        Provisioned
        
        automatic scaling of throughput
        
        automatic scaling of storage
        
        Change tracking with triggers
        (integrate with Lambda)
      - Enterprise ready
        
        ACID transactions
        
        Encryption at rest
        
        Point-in-time recovery
        
        .
        
        On-demand backup and restore
        
        back up tables without affecting performance or availability
        
        Corporate/governmental regulatory requirements
    - - What is Amazon DynamoDB?
        
        DynamoDB is a fast and flexible nonrelational database service for any scale.
        
        --> offload the administrative burdens of operating and scaling
      - Consistency model?
        
        Eventually consistent reads (the default) – The eventual consistency option maximizes your read throughput. However, an eventually consistent read might not reflect the results of a recently completed write. All copies of data usually reach consistency within a second. Repeating a read after a short time should return the updated data.
        
        Strongly consistent reads — In addition to eventual consistency, DynamoDB also gives you the flexibility and control to request a strongly consistent read if your application, or an element of your application, requires it. A strongly consistent read returns a result that reflects all writes that received a successful response before the read.
  - - - What is Amazon Redshift?
        Amazon Redshift is a fast, fully managed data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing Business Intelligence (BI) tools
      - Redshift Spectrum
        
        directly run SQL queries against exabytes of unstructured data in Amazon S3. No loading or transformation is required, and you can use open data formats
        
        use open data formats, including Avro, CSV, Grok, Ion, JSON, ORC, Parquet, RCFile, RegexSerDe, SequenceFile, TextFile, and TSV
      - querying capabilities
        
        fast querying capabilities over structured data using familiar SQL-based clients and business intelligence (BI) tools
        
        JDBC connections
        
        ODBC connections
      - Data Integration and Loading
        
        Data sources
        
        S3
        
        DynamoDB
        
        EMR
        
        SSH-enabled host on:
        
        EC2
        
        on-premises
        
        Tools
        
        Data Pipeline
        
        Import/Export
        
        Glue
      - FAQ
      - Features
  - - - General
        
        What is Amazon Aurora?
        
        relational database engine
        
        MySQL compatible
        
        Certain MySQL features like the MyISAM storage engine are not available with Amazon Aurora
        
        wire-compatible with MySQL 5.6 and 5.7 using the InnoDB storage engine.
        
        PostgreSQL compatible
        
        supports the same set of PostgreSQL extensions that are supported with RDS for PostgreSQL 9.6 and 10, making it easy to move applications between the two engines
        
        wire-compatible with PostgreSQL 9.6 and 10, and
        
        Performance
        
        five times the performance of MySQL without requiring any changes to most MySQL applications
        
        significant increases over MySQL performance by tightly integrating the database engine with an SSD-based virtualized storage layer purpose-built for database workloads, reducing writes to the storage system, minimizing lock contention and eliminating delays created by database process threads
        
        three times the performance of PostgreSQL
        
        significant increases over PostgreSQL performance by tightly integrating the database engine with an SSD-based virtualized storage layer purpose-built for database workloads, reducing writes to the storage system, minimizing lock contention and eliminating delays created by database process threads
        
        RDS manages your Amazon Aurora databases, handling time-consuming tasks
        
        provisioning
        
        patching
        
        backup
        
        recovery
        
        failure detection
        
        repair
        
        How can I migrate from MySQL to Amazon Aurora and vice versa?
        
        standard mysqldump utility to export data from MySQL and mysqlimport utility to import data
        
        RDS’s DB Snapshot migration feature to migrate an RDS MySQL DB Snapshot to Amazon Aurora using the AWS Management Console
        
        What are IOs in Amazon Aurora and how are they calculated?
        
        input/output operations performed by the Aurora database engine against its SSD-based virtualized storage layer
        
        Every database page read operation counts as one IO
      - Hardware and Scaling
        
        minimum storage is 10GB
        
        automatically grow in 10GB increments with no impact to database performance
        
        grows up to 64 TB
      - Backup and Restore
        
        Automated backups are always enabled on Amazon Aurora DB Instances
        
        automatically maintains 6 copies of your data across 3 Availability Zones
        
        if database fails, Aurora will automatically attempt to recover your database in a healthy AZ with no data loss
        
        .
        
        You can share a snapshot with a different AWS account, and the owner of the recipient account can use your snapshot to restore a DB that contains your data.
        
        share data between your various environments (production, dev/test, staging, etc.)
        
        not support sharing automatic DB snapshots. To share an automatic snapshot, you must manually create a copy of the snapshot, and then share the copy
        
        shared Aurora snapshots will only be accessible by accounts in the same region as the account that shares them.
- - - - General
        
        What is Amazon S3 Glacier?
        Amazon S3 Glacier is an extremely low-cost storage service that provides secure, durable, and flexible storage for data backup and archival.
        
        How should I choose between Amazon S3 Glacier and Amazon Simple Storage Service (Amazon S3)?
        Amazon S3 is a durable, secure, simple, and fast storage service designed to make web-scale computing easier for developers. Use Amazon S3 if you need low latency or frequent access to your data. Use Amazon S3 Glacier if low storage cost is paramount, and you do not require millisecond access to your data.
        
        How durable is Amazon S3 Glacier?
        Amazon S3 Glacier is designed to provide average annual durability of 99.999999999% for an archive.
        
        How reliable is Amazon S3 Glacier?
        Glacier is designed for 99.99% availability
        
        What is S3 Glacier Deep Archive?
        S3 Glacier Deep Archive is a new Amazon S3 storage class that provides secure and durable object storage for long-term retention of data that is accessed once or twice in a year.
      - Billing
      - Archives
        
        a durably stored block of information. You store your data in Amazon S3 Glacier as archives. You may upload a single file as an archive, but your costs will be lower if you aggregate your data. TAR and ZIP are common formats that customers use to aggregate multiple files into a single file before uploading to Amazon S3 Glacier
        
        Can range in total size from 1 byte to 40 TB
        
        largest single Upload request is 4 GB
        
        items > 100 MB, consider Multipart upload capability
        
        immutable, i.e. archives can be uploaded and deleted but cannot be edited or overwritten.
      - Vault
        
        Vault Lock
        
        Vault access policies
        
        a way to group archives together in Amazon S3 Glacier. You organize your data in Amazon S3 Glacier using vaults. Each archive is stored in a vault of your choice.
      - Datra retrievals
        
        Data retrieval policies
  - - - General
      - Data Protection and Availability
      - Scale and Performance
      - Provisioned Throughput
      - On-premises Access
        
        How do I copy existing data from on-premises file storage to Amazon EFS?
        
        AWS DataSync
        
        AWS Direct Connect provides a high bandwidth and lower latency dedicated network connection over which you can mount your Amazon EFS file systems. Once mounted, you can use DataSync to copy data into Amazon EFS up to 5x faster than standard Linux copy tools.
      - Storage classes and Lifecycle management
  - - - General
      - Gateway
        
        Volume Gateway
        
        Tape Gateway
        
        File Gateway
      - Hardware Appliance
      - Performance, monitoring and maintenance
  - - - General
        
        What is Amazon S3?
        Amazon S3 is object storage built to store and retrieve any amount of data from anywhere on the Internet. It’s a simple storage service that offers an extremely durable, highly available, and infinitely scalable data storage infrastructure at very low costs.
        
        What can I do with Amazon S3?
        
        Amazon S3 provides a simple web service interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web.
        
        Amazon S3 is also designed to be highly flexible. Store any type and amount of data that you want; read the same piece of data a million times or only for emergency disaster recovery
        
        How much data can I store in Amazon S3?
        
        total volume of data and number of objects you can store are unlimited
        
        Individual Amazon S3 objects size
        
        minimum of 0 bytes
        
        maximum of 5 TB
        
        largest object that can be uploaded in a single PUT is 5 GB
        
        objects > 100 MB: Multipart Upload capability
        
        Storage classes
        
        S3 Standard
        for general-purpose storage of frequently accessed data
        
        99.99% availability
        
        S3 Intelligent-Tiering
        for data with unknown or changing access patterns
        
        S3 Standard-IA
        (Infrequent Access)
        
        99.9% availability
        
        S3 One Zone-IA
        for long-lived, but less frequently accessed data
        
        99.5% availability
        
        S3 Glacier
        
        S3 Glacier Deep Archive
        for long-term archive and digital preservation
      - Billing
        
        Storage Used
        
        volume of storage billed in a month is based on the average storage used throughout the month
        
        includes all object data and metadata stored in buckets that you created under your AWS account.
        
        measure your storage usage in “TimedStorage-ByteHrs,” which are added up at the end of the month to generate your monthly charges
        
        Network Data Transferred Out
        
        For Amazon S3, this charge applies whenever data is read from any of your buckets from a location outside of the given Amazon S3 Region
        
        Data Transfer Out pricing rate tiers take into account your aggregate Data Transfer Out from a given region to the Internet across Amazon EC2, Amazon S3, Amazon RDS, Amazon SimpleDB, Amazon SQS, Amazon SNS and Amazon VPC. These tiers do not apply to Data Transfer Out from Amazon S3 in one AWS Region to another AWS Region.
      - Regions
        
        Data location
        
        You specify an AWS Region when you create your Amazon S3 bucket
        
        For S3 Standard, S3 Standard-IA, and S3 Glacier storage classes, your objects are automatically stored across multiple devices spanning a minimum of three Availability Zones, each separated by miles across an AWS Region.
        
        Objects stored in the S3 One Zone-IA storage class are stored redundantly within a single Availability Zone in the AWS Region you select.
        
        How do I decide which AWS Region to store my data in?
        
        near to your customers, your data centers, or your other AWS resources in order to reduce data access latencies.
        
        remote from your other operations for geographic redundancy and disaster recovery purposes.
        
        enables you to address specific legal and regulatory requirements.
        
        allows you to reduce storage costs. You can choose a lower priced region to save money. For S3 pricing information, please visit the S3 pricing page.
      - Security
        
        Security
        
        secure by default
        
        Upon creation, only the resource owners have access
        
        supports user authentication to control access to data
        
        You can use access control mechanisms such as bucket policies and Access Control Lists (ACLs) to selectively grant permissions to users and groups of users.
        
        security options
        
        SSL endpoint using HTTPS protocol for secure upload/download
        
        Server-Side Encryption (SSE) option to encrypt data stored at rest, if you need extra security
        
        can configure your Amazon S3 buckets to automatically encrypt objects before storing them if the incoming storage requests do not have any encryption information.
        
        own encryption libraries to encrypt data before storing it in Amazon S3 can be used
        
        How to control access to data stored?
        
        IAM policies
        
        enables organizations with multiple employees to create and manage multiple users under a single AWS account.
        
        customers can grant IAM users fine-grained control to their Amazon S3 bucket or objects while also retaining full control over everything the users do
        
        ACLs
        
        grant specific permissions to specific users for an individual bucket or object
        
        READ
        
        WRITE
        
        FULL CONTROL
        
        Query String Authentication
        
        create a URL to an Amazon S3 object which is only valid for a limited time
      - Durability & Data Protection
        
        Durability & Data Protection
        
        Standard,
        Standard–IA,
        One Zone-IA, and
        Glacier are all designed to
        99.999999999% durability of objects over a given year
        
        How?
        
        Standard,
        Standard-IA, and
        Glacier storage classes redundantly store your objects on multiple devices across a
        minimum of three Availability Zones (AZs) in an Amazon S3 Region
        before returning SUCCESS
        
        One Zone-IA storage class stores data
        redundantly across multiple devices within a single AZ
      - Transfer Acceleration
        
        Transfer Acceleration
        
        What is S3 Transfer Acceleration?
        Amazon S3 Transfer Acceleration enables fast, easy, and secure transfers of files over long distances between your client and your Amazon S3 bucket.
        S3 Transfer Acceleration leverages Amazon CloudFront’s globally distributed AWS Edge Locations. As data arrives at an AWS Edge Location, data is routed to your Amazon S3 bucket over an optimized network path.
        
        Can I use S3 Transfer Acceleration with multipart uploads?
        Yes, S3 Transfer Acceleration supports all bucket level features including multipart uploads.
        
        .
        
        How should I choose between S3 Transfer Acceleration and AWS Snow Family (Snowball, Snowball Edge, and Snowmobile)?
        The AWS Snow Family is ideal for customers moving large batches of data at once. The AWS Snowball has a typical 5-7 days turnaround time. As a rule of thumb, S3 Transfer Acceleration over a fully-utilized 1 Gbps line can transfer up to 75 TBs in the same time period. In general, if it will take more than a week to transfer over the Internet, or there are recurring transfer jobs and there is more than 25Mbps of available bandwidth, S3 Transfer Acceleration is a good option. Another option is to use both: perform initial heavy lift moves with an AWS Snowball (or series of AWS Snowballs) and then transfer incremental ongoing changes with S3 Transfer Acceleration.
      - Storage Management
        
        Storage Management
        
        .
        
        .
        
        What is Amazon S3 Cross-Region Replication (CRR)?
        CRR is an Amazon S3 feature that automatically replicates data between AWS Regions.
        With CRR, you can set up replication at a
        -bucket level, a
        -shared prefix level, or an
        -object level using S3 object tags.
        You can use CRR to provide lower-latency data access in different geographic regions. CRR can also help if you have a compliance requirement to store copies of data hundreds of miles apart.
- - - - FAQ
      - Product Details
        
        allows you to define data movement and transformations across various AWS services, as well as for on-premises resources
        
        you define the dependent processes to create your pipeline comprised of the data nodes that contain your data; the activities, or business logic, such as EMR jobs or SQL queries that will run sequentially; and the schedule on which your business logic executes
        
        managed ETL (Extract-Transform-Load) service
  - - - General
        
        What can I do with Amazon Athena?
        Amazon Athena helps you analyze data stored in Amazon S3. You can use Athena to run ad-hoc queries using ANSI SQL, without the need to aggregate or load the data into Athena. Amazon Athena can process unstructured, semi-structured, and structured data sets. Examples include CSV, JSON, Avro or columnar data formats such as Apache Parquet and Apache ORC. Amazon Athena integrates with Amazon QuickSight for easy visualization. You can also use Amazon Athena to generate reports or to explore data with business intelligence tools or SQL clients, connected via an ODBC or JDBC driver.
        
        How do you access Amazon Athena?
        Amazon Athena can be accessed via the AWS Management Console, an API, or an ODBC or JDBC driver. You can programmatically run queries, add tables or partitions using the ODBC or JDBC driver.
        
        What is the underlying technology behind Amazon Athena?
        Amazon Athena uses Presto with full standard SQL support and works with a variety of standard data formats, including CSV, JSON, ORC, Avro, and Parquet.
        
        How does Amazon Athena store table definitions and schema?
        Amazon Athena uses a managed Data Catalog to store information and schemas about the databases and tables that you create for your data stored in Amazon S3. In regions where AWS Glue is available, you can upgrade to using the AWS Glue Data Catalog with Amazon Athena. In regions where AWS Glue is not available, Athena uses an internal Catalog.
        
        Why should I upgrade to AWS Glue Data Catalog?
        AWS Glue is a fully managed ETL service. Glue has three main components: 1) a crawler that automatically scans your data sources, identifies data formats and infers schemas, 2) a fully managed ETL service that allows you to transform and move data to various destinations, and 3) a Data Catalog that stores metadata information about databases & tables either stored in S3 or an ODBC- or JDBC-compliant data store. To use the benefits of Glue, you must upgrade from using Athena’s internal Data Catalog to the Glue Data Catalog.
        The benefits of upgrading to the Glue Data Catalog are:
        
        Unified Metadata Repository: AWS Glue is integrated across a wide range of AWS services. AWS Glue supports data stored in Amazon Aurora, Amazon RDS MySQL, Amazon RDS PostreSQL, Amazon Redshift, and Amazon S3, as well as MySQL and PostgreSQL databases in your Virtual Private Cloud (Amazon VPC) running on Amazon EC2. AWS Glue provides out-of-the-box integration with Amazon Athena, Amazon EMR, Amazon Redshift Spectrum, and any Apache Hive Metastore-compatible application.
        
        Automatic schema and partition recognition: AWS Glue automatically crawls your data sources, identifies data formats, and suggests schemas and transformations. Crawlers can help automate table creation and automatic loading of partitions.
        
        Easy to build pipelines: AWS Glue’s ETL engine generates Python code that is customizable, reusable, and portable. You can edit the code using your favorite IDE or notebook and share it with others using GitHub. Once your ETL job is ready, you can schedule it to run on AWS Glue's fully managed, scale-out Spark infrastructure. AWS Glue is serverless, so it handles provisioning, configuration, and scaling of the resources required to run your ETL jobs, allowing you to tightly integrate ETL in your workflow.
        
        What is Amazon Athena?
        Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to setup or manage, and you can start analyzing data immediately. You don’t even need to load your data into Athena, it works directly with data stored in S3. To get started, just log into the Athena Management Console, define your schema, and start querying. Amazon Athena uses Presto with full standard SQL support and works with a variety of standard data formats, including CSV, JSON, ORC, Apache Parquet and Avro. While Amazon Athena is ideal for quick, ad-hoc querying and integrates with Amazon QuickSight for easy visualization, it can also handle complex analysis, including large joins, window functions, and arrays.
      - Athena vs. other big data services
        
        Athena vs. EMR vs. Redshift
        
        Redshift provides the fastest query performance for enterprise reporting and business intelligence workloads, particularly those involving extremely complex SQL with multiple joins and sub-queries.
        
        EMR makes it simple and cost effective to run highly distributed processing frameworks such as Hadoop, Spark, and Presto when compared to on-premises deployments. Amazon EMR is flexible - you can run custom applications and code, and define specific compute, memory, storage, and application parameters to optimize your analytic requirements.
        
        Athena provides the easiest way to run ad-hoc queries for data in S3 without the need to setup or manage any servers.
        
        When should you use a full featured enterprise data warehouse, like Amazon Redshift vs. a query service like Amazon Athena?
        
        A data warehouse like Amazon Redshift is your best choice when you need to pull together data from many different sources – like inventory systems, financial systems, and retail sales systems – into a common format, and store it for long periods of time, to build sophisticated business reports from historical data; then a data warehouse like Amazon Redshift is the best choice.
        
        Data warehouses collect data from across the company and act as the “single source of truth” for report generation and analysis. Data warehouses pull data from many sources, format and organize it, store it, and support complex, high speed queries that produce business reports. The query engine in Amazon Redshift has been optimized to perform especially well on this use case - where you need to run complex queries that join large numbers of very large database tables. TPC-DS is a standard benchmark designed to replicate this use case, and Redshift runs these queries up to 20x faster than query services that are optimized for unstructured data. When you need to run queries against highly structured data with lots of joins across lots of very large tables, you should choose Amazon Redshift.
        
        By comparison, query services like Amazon Athena make it easy to run interactive queries against data directly in Amazon S3 without worrying about formatting data or managing infrastructure. For example, Athena is great if you just need to run a quick query on some web logs to troubleshoot a performance issue on your site. With query services, you can get started fast. You just define a table for your data and start querying using standard SQL.
        
        You can also use both services together. If you stage your data on Amazon S3 before loading it into Amazon Redshift, that data can also be registered with and queried by Amazon Athena.
        
        When should I use Amazon EMR vs. Amazon Athena?
        
        When should I use Amazon EMR vs. Amazon Athena?
        
        You should use Amazon Athena if you want to run interactive ad hoc SQL queries against data on Amazon S3, without having to manage any infrastructure or clusters.
        
        EMR goes far beyond just running SQL queries. With EMR you can run a wide variety of scale-out data processing tasks for applications such as machine learning, graph analytics, data transformation, streaming data, and virtually anything you can code. You should use Amazon EMR if you use custom code to process and analyze extremely large datasets with the latest big data processing frameworks such as Spark, Hadoop, Presto, or Hbase. Amazon EMR gives you full control over the configuration of your clusters and the software installed on them.
        
        Can I use Amazon Athena to query data that I process using Amazon EMR?
        
        Yes, Amazon Athena supports many of the same data formats as Amazon EMR. Athena’s data catalog is Hive metastore compatible. If you're using EMR and already have a Hive metastore, you simply execute your DDL statements on Amazon Athena, and then you can start querying your data right away without impacting your Amazon EMR jobs.
  - - - Glue
        
        General
        
        What is AWS Glue?
        AWS Glue is a fully-managed, pay-as-you-go, extract, transform, and load (ETL) service that automates the time-consuming steps of data preparation for analytics.
        
        runs the ETL jobs on a fully managed, scale-out Apache Spark environment to load your data into its destination
        
        automatically discovers and profiles your data via the Glue Data Catalog
        
        recommends and generates ETL code to transform your source data into target schemas
        
        Main components
        
        Data Catalog which is a central metadata repository
        
        How do I get my metadata into the AWS Glue Data Catalog?
        
        1 more item...
        
        you can add and update table details manually by using the AWS Glue Console or by calling the API. You can also run Hive DDL statements via the Amazon Athena Console or a Hive client on an Amazon EMR cluster.
        
        if you already have a persistent Apache Hive Metastore, you can perform a bulk import of that metadata into the AWS Glue Data Catalog by using our import script.
        
        ETL engine that can automatically generate Scala or Python code
        
        How can I customize the ETL code generated by AWS Glue?
        
        AWS Glue’s ETL script recommendation system generates Scala or Python code. It leverages Glue’s custom ETL library to simplify access to data sources as well as manage job execution
        
        You can write ETL code using AWS Glue’s custom library or write arbitrary code in Scala or Python by using inline editing via the AWS Glue Console script editor, downloading the auto-generated code, and editing it in your own IDE
        
        You can also start with one of the many samples hosted in our Github repository and customize that code
        
        How can I build end-to-end ETL workflow using multiple jobs in AWS Glue?
        
        AWS Glue ETL jobs can either be triggered on a schedule or on a job completion event. Multiple jobs can be triggered in parallel or sequentially by triggering them on a job completion event.
        
        You can also trigger one or more Glue jobs from an external source such as an AWS Lambda function.
        
        flexible scheduler that handles dependency resolution, job monitoring, and retries
        
        When should I use AWS Glue?
        You should use AWS Glue to discover properties of the data you own, transform it, and prepare it for analytics
        
        Glue can automatically discover both structured and semi-structured data stored in your data lake on Amazon S3, data warehouse in Amazon Redshift, and various databases running on AWS
        
        It provides a uinified view of your data via the Glue Data Catalog that is available for ETL, querying and reporting using services like Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum
        
        Glue automatically generates Scala or Python code for your ETL jobs that you can further customize using tools you are already familiar wit
        
        Glue vs. ...
        
        When to choose Glue above another AWS service?
        
        Glue vs. EMR
        
        AWS Glue works on top of the Apache Spark environment to provide a scale-out execution environment for your data transformation jobs. AWS Glue infers, evolves, and monitors your ETL jobs to greatly simplify the process of creating and maintaining jobs. Amazon EMR provides you with direct access to your Hadoop environment, affording you lower-level access and greater flexibility in using tools beyond Spark.
        
        Glue vs. Database Migration
        
        AWS Database Migration Service (DMS) helps you migrate databases to AWS easily and securely. For use cases which require a database migration from on-premises to AWS or database replication between on-premises sources and sources on AWS, we recommend you use AWS DMS. Once your data is in AWS, you can use AWS Glue to move and transform data from your data source into another database or data warehouse, such as Amazon Redshift.
        
        Glue vs. Batch
        
        AWS Batch enables you to easily and efficiently run any batch computing job on AWS regardless of the nature of the job. AWS Batch creates and manages the compute resources in your AWS account, giving you full control and visibility into the resources being used. AWS Glue is a fully-managed ETL service that provides a serverless Apache Spark environment to run your ETL jobs. For your ETL use cases, we recommend you explore using AWS Glue. For other batch oriented use cases, including some ETL use cases, AWS Batch might be a better fit.
        
        Glue vs. Kinesis
        
        Amazon Kinesis Data Analytics allows you to run standard SQL queries on your incoming data stream. You can specify a destination like Amazon S3 to write your results. Once your data is available in your target data source, you can kick off an AWS Glue ETL job to do further transform your data and prepare it for additional analytics and reporting.
        
        Glue vs. Data Pipeline
        
        Furthermore, AWS Glue ETL jobs are Scala or Python based. If your use case requires you to use an engine other than Apache Spark or if you want to run a heterogeneous set of jobs that run on a variety of engines like Hive, Pig, etc., then AWS Data Pipeline would be a better choice.
        
        AWS Data Pipeline provides a managed orchestration service that gives you greater flexibility in terms of the execution environment, access and control over the compute resources that run your code, as well as the code itself that does data processing. AWS Data Pipeline launches compute resources in your account allowing you direct access to the Amazon EC2 instances or Amazon EMR clusters.
        
        Pricing and billing
        
        Pricing and billing
        
        Additionally, you will pay an hourly rate, billed per second, for the ETL job and crawler run, with a 10-minute minimum for each
        
        If you choose to use a development endpoint to interactively develop your ETL code, you will pay an hourly rate, billed per second, for the time your development endpoint is provisioned, with a 10-minute minimum.
        
        You will pay a simple monthly fee, above the AWS Glue Data Catalog free tier, for storing and accessing the metadata in the AWS Glue Data Catalog
  - - - Use cases
        
        Clickstream analysis
        
        Real-time analytics
        
        Log analysis
        
        Extract Transform Load (ETL)
        
        Predictive Analytics
        
        Genomics
      - Definitions
        
        What is Apache Hadoop?
        Apache Hadoop is an open source framework that is used to efficiently store and process large datasets ranging in size from gigabytes to petabytes of data. Instead of using one large computer to store and process the data, Hadoop allows clustering multiple computers to analyze massive datasets in parallel more quickly. You can learn more about Apache Hadoop here.
        
        What is Amazon EMR?
        Amazon EMR is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).
        
        What is Apache Spark?
        Apache SparkTM is an open-source, distributed processing system used for big data workloads. It utilizes in-memory caching, and optimized query execution for fast analytic queries against data of any size. Amazon EMR is the best place to deploy Apache Spark in the cloud, because it combines the integration and testing rigor of commercial Hadoop & Spark distributions with the scale, simplicity, and cost effectiveness of the cloud. It allows you to launch Spark clusters in minutes without needing to do node provisioning, cluster setup, Spark configuration, or cluster tuning. Learn more about Spark and Spark on Amazon EMR.
        
        What is Presto?
        Presto (or PrestoDB) is an open source, distributed SQL query engine, designed from the ground up for fast analytic queries against data of any size. With Amazon EMR, you can launch Presto clusters in minutes without needing to do node provisioning, cluster setup, Presto configuration, or cluster tuning. EMR enables you to provision one, hundreds, or thousands of compute instances in minutes. Learn more about Presto and Presto on Amazon EMR.
        
        What is a cluster step?
        A cluster step is a user-defined unit of processing, mapping roughly to one algorithm that manipulates the data. A step is a Hadoop MapReduce application implemented as a Java jar or a streaming program written in Java, Ruby, Perl, Python, PHP, R, or C++. For example, to count the frequency with which words appear in a document, and output them sorted by the count, the first step would be a MapReduce application which counts the occurrences of each word, and the second step would be a MapReduce application which sorts the output from the first step based on the counts.
      - General
        Easily Run and Scale Apache Spark, Hadoop, HBase, Presto, Hive, and other Big Data Frameworks
        
        General
        Easily Run and Scale Apache Spark, Hadoop, HBase, Presto, Hive, and other Big Data Frameworks
        
        .
        
        What can I do with Amazon EMR?
        
        Using Amazon EMR, you can instantly provision as much or as little capacity as you like to perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research. Amazon EMR lets you focus on crunching or analyzing your data without having to worry about time-consuming set-up, management or tuning of Hadoop clusters or the compute capacity upon which they sit.
        
        Amazon EMR is ideal for problems that necessitate the fast and efficient processing of large amounts of data. The web service interfaces allow you to build processing workflows, and programmatically monitor progress of running clusters. In addition, you can use the simple web interface of the AWS Management Console to launch your clusters and monitor processing-intensive computation on clusters of Amazon EC2 instances.
        
        .
        
        What can I do with Amazon EMR that I could not do before?
        
        Amazon EMR significantly reduces the complexity of the time-consuming set-up, management. and tuning of Hadoop clusters or the compute capacity upon which they sit. You can instantly spin up large Hadoop clusters which will start processing within minutes, not hours or days. When your cluster finishes its processing, unless you specify otherwise, it will be automatically terminated so you are not paying for resources you no longer need.
        
        Using this service you can quickly perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research.
        
        As a software developer, you can also develop and run your own more sophisticated applications, allowing you to add functionality such as scheduling, workflows, monitoring, or other features.
        
        What is the data processing engine behind Amazon EMR?
        Amazon EMR uses Apache Hadoop as its distributed data processing engine. Hadoop is an open source, Java software framework that supports data-intensive distributed applications running on large clusters of commodity hardware. Hadoop implements a programming model named “MapReduce,” where the data is divided into many small fragments of work, each of which may be executed on any node in the cluster. This framework has been widely used by developers, enterprises and startups and has proven to be a reliable software platform for processing up to petabytes of data on clusters of thousands of commodity machines.
        
        Who can use Amazon EMR?
        Anyone who requires simple access to powerful data analysis can use Amazon EMR. You don’t need any software development experience to experiment with several sample applications available in the Developer Guide and on the AWS Big Data Blog.