Please enable JavaScript.
Coggle requires JavaScript to display documents.
Migration - Coggle Diagram
Migration
Storage Gateway
Types
Volume Gateway
Servers ---(iSCSI)--> Volume Gateway ---(HTTPs)--> AWS Storage Gateway --> S3 --> AWS Backup --> EBS Snapshot
- iSCSI mount for any mode
- Max 32 volumes per gateway appliace
- Stored Volumes: store all you data locally and use AWS storage for asynchronous backup (store periodic point-in-time backups - snapshots - in AWS)
- Up to 16 TB per volume for a maximum of 512 TB per gateway (32 volumes, each 16 TB in size)
- Cached Volumes: S3 is your primary storage. Only frequently accessed data is cached locally on the Volume Gateway appliance
- Up to 32 TB per volume for a maximum of 1 PB per gateway (32 volumes, each 32 TB in size)
- Volumes are in S3 bucket managaed by Storage Gateway (cannot access with S3 API)
- You can take point-in-time snapshots of volumes then made available as EBS snapshots, convert into Storage Gateway Volumes or EBS Volumes
- Use cases: backup on-prem to AWS, migration of volumes to the cloud, DR to the cloud
Tape Gateway
Backup Server ---(iSCSI VTL)--> Tape Gateway ---(HTTPs)--> AWS Storage Gateway --> S3 ---(Eject from backup app)---> S3 Glacier
- Replaces physical tapes
- Do not change current workflow
- Integrates with bespoke backup technologies (NetBackup, Veem, Backup Exec, ...)
- Files are not individually accessible in S3 for recovery
- Use Cases: backup on-prem to AWS, replace physical tape libraries
File Gateway
Servers ---(NFS/SMB)--> S3 File Gateway ---(HTTPs)--> AWS Storage Gateway --> S3 (no S3 Glacier) ---(lifecycle)--> any S3 classes
Servers ---(SMB)--> FSx File Gateway ---(HTTPs)--> AWS Storage Gateway --> FSx for Windows (all features) ---(backup)--> any S3 classes
- Keep a local cache of recently used files
- Extend on-premise storage
- Helps with migrations to AWS
- It is an option to access S3 via filesystem
- S3 File Gateway
- S3 location: S3 bucket, S3 Access Point, S3 Access Point Alias
- You can specify the prefix
- You can use S3 VPC Endpoint
- Automated cache refresh from S3 enable setting TTL
- File upload notification
- SMB Protocol has integration with Active Directory (AD)
- FSx File Gateway
- Service Endpoint: Public, VPC and FIPS (validated cryptographic modules)
- Gateway connection option: IP vs Activation Key
- Integrate with Active Directory
- FSx for Windows can be directly accessed via SMB from on-premise clients, but FSX File Gateway add caching improving performance
- Use Cases: backup on-prem to AWS, shift on-premise storage to cloud-backed file shares, low-latency access your on-prem app to cloud data
Exam Tips
- Think of this service anytime the question mentions on-premise storage, which Storage Gateway type can complement existing architecture
- On-premise running out of space is solved with cached File Gateway
Hybrid cloud storage service to merge on-premise resources with AWS. It can help in both:
- one-time migration
- long-term pairing of your architecture with AWS
It comes in the form of VM(s) for VMware, Hyper-V, KVM, EC2 or Hardware Appliance. Picks data locally and ships to AWS
Use Cases:
1 Migration - move backups to AWS
2 Modernization - shift on-premise storage to cloud-backed file shares
3 Reinvent - low-latency access your on-prem app to cloud data
Manage Volume Gateway
- Upload buffer - a staging area for the data before the gateway uploads data to S3. Your gateway uploads this buffer data over SSL connection to AWS
- Cache storage - acts as the on-premises durable store for data that is pending upload to S3 from the upload buffer. When your application performs I/O on a volume or tape, the gateway saves the data to the cache storage for low-latency access. When your application requests data from a volume or tape, the gateway first checks the cache storage for the data before downloading the data from AWS.
- Use different disks for cache and upload buffer. If using RAID, ensure that cache and upload buffer disks use separate RAID controllers at the hardware level
- Add at least 2 different upload buffer disks
- Use a RAID 0 striped raid configuration for cache + upload buffer devices to improve throughput. This is especially critical for the cache disk
Configure additional upload buffer or cache storage:
- You can add storage capacity to your gateway without interrupting functionality or causing downtime
- When adding cache or upload buffer to an existing gateway:
- You must create new disks on the gateway
- Do not remove or change the size of existing disks that have already been allocated as cache or upload buffer
- Create new disks in your host (hypervisor or Amazon EC2 instance)
- Choose either UPLOAD BUFFER or CACHE STORAGE from the Allocated to drop-down menu
Add a Volume:
- As your application needs grow, you might need to add more volumes to your gateway
- The gateway must have sufficient buffer and cache space for new volumes
- up to 2 TiB of upload buffer capacity for each gateway
- set cache storage at 1.1 times the upload buffer size
Expanding the Size of a Volume - one of the following:
- Create a snapshot of the volume you want to expand and then use the snapshot to create a new volume of a larger size
- Use the cached volume you want to expand to clone a new volume of a larger size
-
-
AWS DataSynch
On-Premise Servers ---(NFS/SMB/S3API/HDFS)--> AWS DataSynch agent ---(TLS)--> AWS DataSynch --> S3, EFS, FSx
Exam Tips
- DataSynch is more for one-time migration, if you have an hybrid architecture and need a continuous thus Storage Gateway is better
- While DataSync usually uses an agent, an agent is not required when transferring between AWS storage services in the same AWS account
- Agent-based solution (deployed as VM) for one-time data migration from on-premise to AWS
- Move data from NFS / SMB / HDFS / S3API to AWS Storage solutions: S3, EFS and FSx
- Transfer data between on-premises and AWS
- Transfer data between AWS storage services
- Transfer data between AWS and other locations (other public clouds)
Replication tasks can be scheduled hourly, daily, weekly
- File permissions and metadata are preserved (NFS POSIX, SMB...)
- It is compliant to NFS POSIX and SMB permissions
One agent task can use 10 Gbps, can setup a bandwidth limit
-
-
AWS Snow family
Snowball Edge
- 48TB to 81TB storage
- Flavors:
- Storage Optimized -> 80TB of HDD for block storage or S3-compatible (now also 210TB option since May 2023)
- Compute Optimized -> 42TB or 28TB NVMe for block storage or S3-compatible
- with GPU -> identical to compute optimized + GPUs
- Varying amount of CPU and RAM
- Off-the-grid computing or migration to AWS
Snowmobile
- 100PB storage
- Exabyte-scale data center migration
- Better than Snowball if you transfer more than 10 PB
Snowcone
- 4GB memory and 2 vCPUs
- Snowcone: 8TB storage HDD
- Snowcone SDD: 14TB SDD
- Migrate data after you have processed it
- IoT sensor integration
- Edge computing where space and power are constrained
- Can be sent back to AWS offline (data transfer use case), or connect it to internet and use AWS DataSync to send data (edge compute use case)
Exam Tips:
- Scenario questions will ask you to pick a solution for data migration, make sure to pay attention to restrictions (no internet access, large amount of data, slow internet)
- Use Cases: be aware of Snow family different use cases
- Both Ways: move data to and from AWS
- Timing: generally turnaround is a week, but mostly depends on customer
- Consider Snow Family if network transfer takes more than a week
A set of secure appliances that provides both storage as well as compute capabilities that can operate in remote locations that do not have data center or reliable networking
HopsHub for Snow Family
- GUI you can use to manage your AWS Snowball devices
- Unlock devices, Deploy edge computing workloads and Execute data migration
- Configure single or clustered devices
- SW you download and install on your Windows or Mac laptop
Improving Transfer Performance
- From most impactful to least
- Perform multiple write operations at one time - from multiple terminals
- Transfer small files in batches – files =< 1MB should be batched, user TAR, AIP and tar.gz to auto-extract if <100GB, up 10K files per batch
- Don't perform other operations on files during transfer
- Reduce local network use
- Eliminate unnecessary hops – directly connect to the computer
- File interface typically 25MB/s ÷ 40 MB/s. If not enough, use the NFS adapter. The NFS adapter typically 250 MB/s ÷ 400 MB/s
- S3 Adapter for Snowball typically 250 MB/s ÷ 400 MB/s (do not use file interface and S3 adapter at the same time on the same bucket)
AWS Migration Hub
Server Migration Service
- Schedule the upload of VMware VMDK to S3 buckets
- Convert VMDK to EBS Snapshot
- Create AMI from EBS Snapshot
- Launch EC2 instances from AMI
Database Migration Service
- Step 1: Pick an Oracle DB (SQL Server) and run through the AWS Schema Converter Tool and applies output to an Aurora Database (or another database)
- Step 2: Pick a database anywhere (on-premise, EC2, RDS) and move to an Aurora Database. It can consolidates multiple DBs in to a single pice of architecture
Single place (pane-of-glass) to track progress of your app migration to AWS. Integrated with the services that do the heavy-lift SMS (Server Migration Service) and DMS (Database Migration Service)
Exam Tips
- There is no app migration DIY
- In the exam these migrations tool will solve any app migration needs
- When you see Oracle or SQL Server you really want to move to Aurora or another DB type
- Focus on answers that include doing a complete migration
Phases
Discover
- Import Data
- Deploy AWS Discovery Connector (to VMware vCenter)
- Deploy AWS Discovery Agent (into VMs)
-
Migrate
- Connect migration tools (AWS and third party tools)
- Migrate using connected tools
-
-
-
AWS Transfer family
- Allows to move files from/to S3 or EFS using:
- FTP: File Transfer Protocol (FTP)
- FTPS: File Transfer Protocol over SSL (FTPS)
- SFTP: Secure File Transfer Protocol (SFTP)
- Store and manage users’ credentials within the service
- Integrates with authentication systems: LDAP, AD, Cognito ...
Exam Tips
- Good when you have legacy app/users using protocols that cannot be changed
- The DNS entry stays the same, but the location for file storage becomes S3
-
Endpoint Types
- Public Endpoint
- Accessible from internet
- IP can change and managed by AWS
- better to use DNS Name for the endpoint
- Can't setup allow list by source IPs
- VPC Endpoint with Internal Access
- Accessible only from VPC
- Static private IPs
- Setup allow list (SG & NACL)
- VPC Endpoint with Internet-facing Access
- Accessible from both Internet and VPC
- Static private IPs
- Static public IPs (Elastic IPs)
- Setup SG
Moving Data to AWS
Direct Connect - This can be faster and more secure, but it's not always practical if it's not needed after the migration
Physical - Put you data in transportable storage equipments and deliver it to AWS. This bypasses internet
Internet - Using existing connection is convenient but potentially very slow and could be a security risk
-
-
-
AWS Migration Evaluator
- Helps you build a data-driven business case for migration to AWS
- Provides a clear baseline of what your organization is running today
- Data:
- Install Agentless Collector
- Import Data from from 3rd party tools or Application Discovery Service
- Review Quick Insights report for customized cost insights
- Business Case gain expert guidance to build a business case for migration to AW
The 6R
- Rehosting: “lift and shift”
- Simple migrations by re-hosting on AWS (applications, databases, data...)
- No cloud optimizations being done, application is migrated as is
- Save as much as 30% on cost
- Example: Migrate using AWS VM Import/Export, AWS Server Migration Service
- Replatforming
- Not changing the core architecture
- Leverage some cloud optimizations
- Example: migrate your database to RDS
- Example: migrate your application to Elastic Beanstalk (Java with Tomcat)
- Repurchase: “drop and shop”
- Moving to a different product while moving to the cloud
- Often moving to SaaS
- Expensive in the short term, but quick to deploy
- Example: CRM to Salesforce.com, HR to Workday, CMS to Drupal
- Refactoring / Re-architecting
- Reimagining how the application is architected using Cloud Native features
- Driven by the need of the business to add features, scale, performance (current architecture has constraints)
- Tends to be the most expensive, but, if you have a good product-market fit, it can also be the most beneficial
- Example: move an application to Serverless architectures, use AWS S3
- Retire
- Turn off things you don’t need (maybe as a result of Re-architecting)
- Reduce the surface areas for attacks (more security)
- Save cost, maybe up to 10% to 20%
- Focus your attention on resources that must be maintained
- Retain
- Do nothing for now (for simplicity, cost reason, importance...)
- It’s still a decision to make in a Cloud Migration