Coggle requires JavaScript to display documents.
Storage Class Standard (default) One Zone Automatic backups - Automatically backup your file system data with AWS Backup using recommended settings Lifecycle management - EFS Intelligent-Tiering uses Lifecycle Management to automatically achieve the right price and performance blend for your application by moving your files between the Standard and Standard-Infrequent Access storage classes Transition into IA- Transition files from Standard to Standard-Infrequent Access (None,1, 7,14,30, 60, 90) Transition out of IA- Transition files from Standard-Infrequent Access to Standard (On first access) Encryption - Choose to enable encryption of your file system's data at rest. Uses the AWS KMS service key (aws/elasticfilesystem) by default Optionally you can specify/create your own KMS key Throughput mode Bursting - Provides throughput that scales with the amount of storage for workloads with basic performance requirements Enhanced - Provides more flexibility and higher throughput levels for workloads with a range of performance requirements Elastic (Recommended) - Use this mode for workloads with unpredictable I/O. With Elastic mode, your throughput scales automatically and you only pay for what you use Provisioned- Use this mode if you can estimate your workload's throughput requirements. With Provisioned mode, you configure your file system's throughput and pay for throughput provisioned Provisioned Throughput (MiB/s) Maximum Read Throughput (MiB/s) Performance mode General Purpose (Recommended) - Ideal for a variety of diverse workloads, including high performance and latency-sensitive applications Max I/O - Designed for highly parallelized workloads that can tolerate higher latencies Network VPC Mount targets - A mount target provides an NFSv4 endpoint at which you can mount an Amazon EFS file system. We recommend creating one mount target per Availability Zone (includes selecting Security Group) Policy options - Select one or more of these common policy options, or create a custom policy using the editor: Prevent root access by default (Identity-based policies can override these default permissions) Enforce read-only access by default (Identity-based policies can override these default permissions) Prevent anonymous access Enforce in-transit encryption for all clients
Standard (default) One Zone
Transition into IA- Transition files from Standard to Standard-Infrequent Access (None,1, 7,14,30, 60, 90) Transition out of IA- Transition files from Standard-Infrequent Access to Standard (On first access)
Optionally you can specify/create your own KMS key
Bursting - Provides throughput that scales with the amount of storage for workloads with basic performance requirements Enhanced - Provides more flexibility and higher throughput levels for workloads with a range of performance requirements Elastic (Recommended) - Use this mode for workloads with unpredictable I/O. With Elastic mode, your throughput scales automatically and you only pay for what you use Provisioned- Use this mode if you can estimate your workload's throughput requirements. With Provisioned mode, you configure your file system's throughput and pay for throughput provisioned Provisioned Throughput (MiB/s) Maximum Read Throughput (MiB/s)
Elastic (Recommended) - Use this mode for workloads with unpredictable I/O. With Elastic mode, your throughput scales automatically and you only pay for what you use Provisioned- Use this mode if you can estimate your workload's throughput requirements. With Provisioned mode, you configure your file system's throughput and pay for throughput provisioned Provisioned Throughput (MiB/s) Maximum Read Throughput (MiB/s)
Provisioned Throughput (MiB/s) Maximum Read Throughput (MiB/s)
General Purpose (Recommended) - Ideal for a variety of diverse workloads, including high performance and latency-sensitive applications Max I/O - Designed for highly parallelized workloads that can tolerate higher latencies
VPC Mount targets - A mount target provides an NFSv4 endpoint at which you can mount an Amazon EFS file system. We recommend creating one mount target per Availability Zone (includes selecting Security Group)
Prevent root access by default (Identity-based policies can override these default permissions) Enforce read-only access by default (Identity-based policies can override these default permissions) Prevent anonymous access Enforce in-transit encryption for all clients
Using the EFS mount helper Using the NFS client
Using the NFS client (select AZ)
sudo yum install -y amazon-efs-utils sudo mkdir efs sudo mount -t efs -o tls <YOU_EFS_NAME>:/ efs (copied from EFS Attach page, the efs at the end is the name of your folder) df -T (to check it)
EFS One Zone class you can only create a single mount target in the same AZ of your EFS EFS Multi AZ Standard Class you can create a mount point in each AZ of the region If your EC2 in AZ 1 access EFS mount point into AZ 2 you will incur into data transfer costs
Query log files stored in S3 such as ELB logs, S3 server access logs, etc. Analyze AWS CUR Generate business reports on data stored in S3 Run query on click-stream data
You need to create a bucket to store Athena query results (Athena > Query editor > Settings) In the Query editor run a query to create a Database "CREATE DATABASE athenadb" In the Query editor run a query to create a Table In the Query editor you can now run your SQL query against your database/table
Real-time analytics Log analytics Application monitoring Security analytics Business data analytics
hardware provisioning, configuring the Elasticsearch cluster software installation and patching failure recovery, automated backups and monitoring failure recovery is the ability to detect a failure in nodes and replace the node ability to scale cluster with a single API call or few click in the AWS Console ability to store up to 3PB of data replicate data across AZ for HA
failure recovery is the ability to detect a failure in nodes and replace the node
a OpenSearch Service Domain or a Cluster the Service Domain is made up by a number of EC2 instances Master Nodes are responsible for: manage the cluster monitor the health of the nodes in the cluster maintain routing information update the cluster state after changes 3 dedicated master nodes is the recommended number (1 not allowed, 2 doesn't provide quorum nodes, 4 is not better than 3) Data Nodes are responsible for: store data in shards performs search query requests CRUD operations
manage the cluster monitor the health of the nodes in the cluster maintain routing information update the cluster state after changes 3 dedicated master nodes is the recommended number (1 not allowed, 2 doesn't provide quorum nodes, 4 is not better than 3)
store data in shards performs search query requests CRUD operations
compatible with open-source Elasticsearch API integrates with Logstash (ingestion) and Kibana (visualization) integrates with CloudWatch for monitoring use CloudWatch Logs, Kinesis Data FireHose and Lambda to ingest data (Lambda is tipically used to respond events in S3 or DynamoDB and store data in OpenSearch Service Domain)
Require HTTPS for all traffic to the domain Node-to-node encryption Encryption at rest after you enable encryption of data at rest, you can’t disable it
Shards distribute your workload across the data nodes in your OpenSearch Service domain When you send data to OpenSearch, you send that data to an index. An index is analogous to a database table, with documents as the rows, and fields as the columns When you create the index, you tell OpenSearch how many primary shards you want to create The primary shards are independent partitions of the full dataset OpenSearch Service automatically distributes your data across the primary shards in an index You can also configure replicas of the index. Each replica shard comprises a full set of copies of the primary shards for that index You should always use at least one replica. Additional replicas provide additional redundancy and read capacity
HA > 99,99% consider using two domains across multiple Regions Small or slowly changing datasets: set up cross-cluster replication to maintain an active-passive model only the leader domain is written to, but either domain can be read from Larger data sets and quickly changing data: configure dual delivery in your ingest pipeline data is written independently to both domains in an active-active model
set up cross-cluster replication to maintain an active-passive model only the leader domain is written to, but either domain can be read from
configure dual delivery in your ingest pipeline data is written independently to both domains in an active-active model
Requires fine-grained access control Lets you use third-party identity providers to log in to Dashboards only for accessing OpenSearch Dashboards through a web browser SAML doesn't require direct communication between your identity provider and your service provider When OpenSearch hosted in a private VPC, you can still use SAML as long as your browser can communicate with both your OpenSearch cluster and your identity provider Supports providers that use the SAML 2.0 standard: Okta, Keycloak, ADFS, Auth0, AWS IAM Identity Center
Okta, Keycloak, ADFS, Auth0, AWS IAM Identity Center
Easy create - quickly create an OpenSearch domain using 'Multi-AZ with Standby' for high availability. You can change some configuration options after the domain is created Standard create - create an OpenSearch domain with your preferred configuration. You can choose all configuration options, including AZ(s), instances and storages, and security configurations Production - use for production workloads for high-availability and performance. Defaults follow the recommended best practices Dev/test - use for development or testing Amazon OpenSearch Service Deployment Option(s) - select a deployment option that corresponds to the availability goals for your application Domain with standby - nodes in one of the Availability Zone (AZ) are reserved as standby. Automatic failover to standby. Provides 99.99% availability and consistent performance. (Recommended) 3-AZ having Active: 2 AZ and Standby: 1 AZ Domains with standby enforce a number of best practices like: data node count, master node count and instance type, replica count, security, auto-tune and software update settings Migrating from domain without standby to domain with standby will automatically enable auto-tune Domain without standby - nodes are distributed across AZ(s). Availability will depends on the number of AZs and copies of data 3 AZs - for 3 AZs, recommend instances in multiples of 3 for equal distribution across the AZs 2 AZs - for 2 AZs, you must choose instances in multiples of 2 1AZs - available only for Dev/test deployments Engine options: Version Enable compatibility mode - certain Elasticsearch OSS clients, such as Logstash, check the cluster version before connecting. Compatibility mode sets OpenSearch to report its version as 7.10 so that these clients continue to work Network - choose internet or VPC access. To enable VPC access, we use private IP addresses from your VPC, which provides an inherent layer of security. You control network access within your VPC using security groups. Optionally, you can add an additional layer of security by applying a restrictive access policy. Internet endpoints are publicly accessible. If you select public access, you should secure your domain with an access policy that only allows specific users or IP addresses to access the domain VPC access (recommended) - slect VPC, subnets and SGs Public access Custom endpoint - each OpenSearch Service domain has an auto-generated endpoint, but you can also add a custom endpoint using AWS Certificate Manager (ACM). You can also enable and configure it after your cluster is created Enable UltraWarm to store even more data on Amazon OpenSearch Service. You can economically retain large amounts of data while keeping the same interactive analysis experience Enable cold storage to further reduce storage costs for data you rarely access. To view data in cold storage, you must first move it to warm storage SAML authentication for OpenSearch Dashboards/Kibana Enable SAML authentication Enable Amazon Cognito authentication Access policy - control whether a request is accepted or rejected when it reaches the Amazon OpenSearch Service domain. If you specify an account, user, or role in this policy, you must sign your requests Only use fine-grained access control - Allow open access to the domain Do not set domain level access policy - All requests to the domain will be denied Configure domain level access policy Encryption Require HTTPS for all traffic to the domain - When enabled, your domain only accepts requests over HTTPS Node-to-node encryption - after you enable node-to-node encryption, you can't disable it Enable encryption of data at rest secures indices and automated snapshots. Once enabled encryption of data at rest, you can’t
Production - use for production workloads for high-availability and performance. Defaults follow the recommended best practices Dev/test - use for development or testing Amazon OpenSearch Service
Domain with standby - nodes in one of the Availability Zone (AZ) are reserved as standby. Automatic failover to standby. Provides 99.99% availability and consistent performance. (Recommended) 3-AZ having Active: 2 AZ and Standby: 1 AZ Domains with standby enforce a number of best practices like: data node count, master node count and instance type, replica count, security, auto-tune and software update settings Migrating from domain without standby to domain with standby will automatically enable auto-tune Domain without standby - nodes are distributed across AZ(s). Availability will depends on the number of AZs and copies of data 3 AZs - for 3 AZs, recommend instances in multiples of 3 for equal distribution across the AZs 2 AZs - for 2 AZs, you must choose instances in multiples of 2 1AZs - available only for Dev/test deployments
3-AZ having Active: 2 AZ and Standby: 1 AZ Domains with standby enforce a number of best practices like: data node count, master node count and instance type, replica count, security, auto-tune and software update settings Migrating from domain without standby to domain with standby will automatically enable auto-tune
3 AZs - for 3 AZs, recommend instances in multiples of 3 for equal distribution across the AZs 2 AZs - for 2 AZs, you must choose instances in multiples of 2 1AZs - available only for Dev/test deployments
Version Enable compatibility mode - certain Elasticsearch OSS clients, such as Logstash, check the cluster version before connecting. Compatibility mode sets OpenSearch to report its version as 7.10 so that these clients continue to work
VPC access (recommended) - slect VPC, subnets and SGs Public access
Enable SAML authentication Enable Amazon Cognito authentication
Only use fine-grained access control - Allow open access to the domain Do not set domain level access policy - All requests to the domain will be denied Configure domain level access policy
Require HTTPS for all traffic to the domain - When enabled, your domain only accepts requests over HTTPS Node-to-node encryption - after you enable node-to-node encryption, you can't disable it Enable encryption of data at rest secures indices and automated snapshots. Once enabled encryption of data at rest, you can’t
3 dedicated Master Nodes (this gives you 2 backup nodes in the event of a failure) 3 AZs and deploy in multiple of 3 distributing equally across 3 AZs If you need to scale just add Data Nodes (no need to add more Master Nodes)
read-only - thus you still need to have data node(s) stores data in S3 uses custom optimized nodes, purpose-built on Nitro, to cache, prefetch, and query that data quickly retain up to 3 PB in a single OpenSearch Service cluster no need to have replica for your warm data (because of S3) requires to have dedicated master nodes (these nodes perform cluster management tasks, but do not hold data or respond to data upload requests)
decouples compute resources from storage available within the OpenSearch Service domain through UltraWarm nodes detach indices from UltraWarm while not in use and free up compute to lower costs you can detach and then attach when you need to access the data query the attached cold data with a similar experience and performance as your warm data
S3 EFS [EBS & EFS] Athena OpenSearch Storage Gateway AWS Backup [EBS & EFS]