Table of Contents 

AWS DMS (Amazon Web Services Database Migration Service) is a managed solution for migrating databases to AWS. It allows users to move data from various sources to cloud-based and on-premises data warehouses.

However, users often encounter challenges when using AWS DMS for ongoing data replication and high-frequency change data capture (CDC) processes. 

Here are our 5 key takeaways from this guide:

  • AWS DMS is not suitable for ongoing real-time data updates due to various challenges in replication and high-frequency CDC processes.
  • "Sync lag" is a major issue in using AWS DMS for ongoing replication, causing a ripple effect that can prevent important changes from being reflected in the target database.
  • High-frequency CDC causes stress on the source database, target database, and AWS DMS service, leading to increased latency, lower throughput, and resource consumption.
  • AWS DMS is challenging in terms of scaling during transaction spikes, complicated setup and maintenance, and higher-than-expected operational and labor costs.
  • To address these challenges and costs, closely monitor AWS DMS replication performance, quickly correct sync lag issues, consider using third-party tools or services, and opt for a more reliable alternative.

In this guide, we will discuss the top challenges faced by AWS DMS CDC users and how to overcome them.

Introduction

The Amazon Web Services (AWS) Database Migration Service (DMS) is a fully-managed solution for migrating databases to AWS. According to Amazon Web Services, AWS DMS allows users to quickly and easily move data from a variety of sources—including Oracle, Microsoft SQL Server, MySQL, and other types of operational systems—to cloud-based and on-premises data warehouses. 

AWS further claims that the DMS service is a “flexible, low-cost solution” that allows you to “continuously and easily replicate data to destination databases with minimal downtime and robust error handling and reporting.” However, not all customers experience AWS DMS as a “flexible, low-cost solution.” Many AWS DMS users—especially those who lack data integration skills and large IT budgets—are surprised to find that the AWS DMS service can bring serious challenges and an excessively high cost of total ownership. 

In this guide, we discuss the top challenges for AWS DMS CDC users and how to overcome them. We also show how Integrate.io can serve as a powerful, easy-to-use, and reliable alternative to AWS DMS.

Let's start with a brief overview of AWS DMS challenges.

Overview of AWS DMS Challenges

As its name implies, AWS DMS is excellent for one-time database migrations. However, the platform tends to bring a host of challenges when used for ongoing data replication and high-frequency change data capture (CDC) processes. In other words, AWS DMS is not ideal for ongoing near-real-time data updates.

Unfortunately, because AWS DMS markets itself as a solution for ongoing data replication, users frequently sign up for the service with this use case in mind—only to encounter serious challenges related to sync lag, CDC performance setbacks, scaling trouble during transaction spikes, complicated setup and maintenance, and higher-than-expected operational and labor costs. Even though the cost of the AWS DMS product itself isn’t that expensive, the challenges detailed below cause the actual cost of total ownership to be exorbitantly high—due to the time spent building it, maintaining it, and repairing it when it breaks. Also, organizations incur enormous business impact costs when their AWS DMS instance goes down.

To help you understand, avoid, and overcome these challenges and costs, let’s take a look at each of them in detail.

Sync Lag

Among the variety of challenges that arise when using AWS DMS for ongoing data replication, the first is “sync lag,” which causes ripples of problems throughout the entire data set.

Sync lag refers to a discrepancy that exists between the source database and the target database. It usually results from an interruption or latency in the replication process. This latency can trigger a ripple effect throughout the entire dataset in the target database—which means that important changes in the source database may not be reflected in the target database. 

In serious cases of AWS DMS sync lag, it’s extremely difficult to bring the target database back into sync without manual intervention. The longer this latency persists, the more difficult it is to correct, and the more corrupted the destination database—and the business intelligence derived from it—become.

According to Dataconomy: “The common goal of data replication is to reduce latency to sub-millisecond periods. Pressing the refresh button on a website and waiting for what seems like an eternity (seconds) to see your information refreshed is a scenario we have all experienced. A user’s productivity is reduced by latency. The objective is achieving near-real-time. For whatever use scenario, zero time lag is the new ideal.”

To overcome AWS DMS CDC sync lag hurdles, the following strategies may help:

  • Closely monitor the performance of the AWS DMS replication process at all times.
  • Establish a plan for quickly and efficiently correcting sync lag issues when they arise. 
  • Consider using tools or third-party services for detecting and correcting sync lag. 

Performance Issues

Change Data Capture (CDC) is the process of tracking the changes in a source database and continually updating those changes to a target database. When the CDC is “high-frequency,” the source database is changing rapidly, and the target database—usually a data warehouse—needs to reflect those changes in near real-time. Although AWS claims that DMS can manage high-frequency CDC scenarios, problems can arise.

Here is why high-frequency CDC creates a problem for AWS DMS:

  • Puts stress on the system: High-frequency CDC puts a lot of stress on the source database, the target database, and the AWS DMS service itself. This can increase latency, lower throughput, and heighten resource consumption. To understand why this is a problem, consider a financial system where stock prices are constantly changing, and a high-frequency CDC process is trying to update stock prices in real time to the target database. Increased latency caused by AWS DMS could cause the wrong prices and values to show, rendering the entire system ineffective. 
  • Higher chances of conflicts and errors in the dataset: When a CDC process needs to make multiple changes to the same database in a short period of time, AWS DMS can struggle to track all of the changes, and the target database could become corrupted and out of sync with the source. Unfortunately, this can render AWS DMS completely ineffective when data accuracy and the timeliness of data replication are critical. For example, organizations using AWS DMS could find it impossible to extract accurate, real-time business intelligence from their data.  

For tech-savvy readers, Alex Souza offers a detailed account of what it's like to carefully tune an AWS DMS instance for high-frequency CDC processes. Souza’s article is a must for anyone trying to overcome the many challenges of using AWS DMS for CDC. According to Souza: 

“With DMS it is possible to configure tasks using the “Change Data Capture” (CDC) technique to capture changes in progress from the source database. For this type of task to work, some settings must be enabled at the source, such as the database logs, as this CDC solution is based on reading the transaction logs and replicating to the destination. This technique is developed in this way to let the consumption of the DMS separate from the transactional, decreasing the chances of any system impact.” 

AWS also claims that users can improve high-frequency CDC processes by carefully tuning their AWS DMS instance. This might involve the following mitigation strategies:

  • Choosing an instance size to match expected workloads.
  • Implementing throttling to limit the volume of data to ensure that the system is capable of handling the load. 
  • Using rate limiting to limit the number of transactions within a given period of time.

Of course, these adjustments could make the entire AWS DMS process more expensive in terms of compute resources, time spent managing the system, and the experienced staff required to make the adjustments. 

Scaling Problems

Another issue we’ve found with AWS DMS is that it can struggle to handle large transaction loads during sudden spikes in traffic. This is particularly frustrating because it can lead to performance issues—such as increased latency, sync lag, timeouts, data loss, and more.

To avoid performance issues during ongoing data replication, AWS DMS users need to apply a considerable amount of manual intervention. Typically, this involves scaling up the AWS infrastructure by adding more compute resources and/or modifying the existing configurations. The following modifications could help in this regard:

  • Increasing buffer sizes
  • Adjusting network throughput
  • Changing replication settings
  • Modifying source and target database settings of either the source or target database
  • Adjusting storage configurations of the source and target databases 

When AWS DMS struggles to manage transaction spikes, load balancing can also help to distribute incoming network traffic across multiple servers or resources to improve the availability, redundancy, and scalability of an application or service. Load balancing procedures may involve directing traffic based on factors such as server health, server capacity, and user sessions. 

Even when your team has the expertise required to implement these manual changes, the process can be time-consuming and costly.

Set Up and Configuration

Aside from the above challenges, one of the most significant barriers to using AWS DMS relates to the skills required to successfully leverage the service. Most businesses and organizations—especially SMEs and smaller organizations—simply cannot afford the skills required to use AWS DMS. For example, AWS DMS generally requires a combination of skills and experience relating to:

  • Database administration skills: This includes an in-depth understanding of database administration and architecture, as well as experience working with relational databases like Oracle, SQL Server, and MySQL.
  • AWS knowledge: Experience with Amazon Web Services (AWS) is key. This includes an understanding of AWS components like Amazon VPC, Amazon S3, and Amazon EC2.
  • Networking knowledge: Knowledge of networking concepts and experience working with virtual private clouds (VPCs) is important, as you may need to set up network security groups, subnets, and VPC peering.
  • Scripting skills: Familiarity with scripting languages like Python or shell scripts is important, as your team may need to write scripts to automate tasks and add customizations to the AWS DMS CDC instance.
  • Data migration experience: Experience with data migration and data replication is another prerequisite.
  • Familiarity with change data capture (CDC): Teams need to have a good understanding of how to set it up, configure, and maintain CDC processes with AWS DMS.

If your team is lacking in these skills, consider paying the extra cost of hiring a third-party AWS DMS consultant. Or, skip AWS DMS altogether and use an alternative data integration service that is easier to use, more reliable, and purpose-built for ongoing data replication. For example, Integrate.io  is made for continuous data replication and high-frequency CDC, but it doesn't require any data integration skills or experience to operate.

Cost

While the cost of using AWS DMS can vary, most organizations can’t afford to use the service or they struggle with the high cost unnecessarily. Here are some of the key cost factors that contribute to the high price of AWS DMS:

  • Data transfer: There may be additional costs associated with data transfer, especially when migrating data across different AWS regions.
  • Processing power: The cost of processing power depends on the size of the data being migrated and the speed of the migration. Some organizations will need to spin up multiple instances of AWS DMS to achieve their high performance and scalability requirements—causing exponential cost increases.
  • Storage: The cost of storage depends on the size of the data being stored in AWS S3 or other storage solutions.
  • Data changes: The cost of data changes depends on (1) the frequency of the changes and (2) the volume of data being changed. 
  • Experienced labor: The cost of labor with required AWS DMS skills and experience is outside of the budget of many SMEs and startups. 

While it’s not possible to estimate the exact cost of using AWS DMS, the AWS Pricing Calculator allows organizations to gain a rough estimate of potential costs. Teams may also want to consult with an experienced AWS DMS expert for an accurate cost assessment.

AWS DMS Alternative

When faced with the difficulties, expenses, high latency, and data quality challenges of using AWS DMS CDC for ongoing data replication and general data integration, many organizations realize that they could benefit from a more reliable, affordable, and easier-to-use alternative. This is where Integrate.io can help to overcome AWS DMS CDC limitations. 

Integrate.io is a powerful data integration platform that anyone can use to quickly and reliably sync data from one system to another. In fact, the drag-and-drop, no-code interface of Integrate.io is so easy to use that a non-tech-savvy team member—without any coding or data integration experience—can create ongoing data replication processes. Without any special training, beginner users can even develop sophisticated data transformation workflows that involve complex schema changes. There is also a robust code editor for those who have the skills and still want the flexibility of coding at their fingertips.

In this way, Integrate.io is built for both data teams and line-of-business "citizen integrators". These citizen integrators (working in business departments like Marketing, CX, and Ops) can use their domain-level expertise to leverage the full power of data more effectively—by creating the right kinds of pipelines to fit their data needs.

The secret to Integrate.io’s ease of use goes further than its no-code interface. Every Integrate.io account comes with unlimited support—by phone, email, messaging, and video conference—from a dedicated data integration specialist. Each account's integration specialist is like a special member of the team, always available to help customize new data connections for unique use cases and scenarios. 

As for its affordability, Integrate.io keeps billing simple. We don’t charge additional amounts based on processing, data volume, or transaction amounts. Instead, we charge an affordable flat rate fee per data connection. This makes monthly billing reliable and predictable—with service arrangements to fit any size budget and any size integration requirement.