Posts

Showing posts from October, 2020

Cutting-edge ETL Tools for AWS DMS

Image
  Businesses today are looking for optimized data warehousing solutions to optimize their database management systems. With organizations being fully data driven now, it is always advisable to migrate databases to advanced platforms like Amazon Web Services (AWS), a cloud provider to fully get the many benefits of it. However, AWS DMS (Database Migration Service) will only be very successful and efficient if the right tools are used. These are called AWS DMS ETL tools and they help to migrate data from different sources to a single database. ETL tools largely facilitate this process.           The components of ETL tools are as follows. ·         Extract (E) – Extracting the required data from Heterogeneous or Homogeneous Data sets. ·         Transform (T) – Transforming or modifying the extracted data to match the data structures supported by the target storage database. ...

Change Data Capture and Microsoft SQL Server Database

Image
  Data security is the prime concern of organizations today, with data breaches, leaks, and hacking becoming a very common occurrence. An important method to ramp up data security is saving changed data in the database so that the historical data is always preserved. For ensuring this, various databases have introduced different solutions over time such as triggers, complex queries, timestamps, and data audits. In 2005, SQL Server introduced “after date”, “after insert”, and “after delete” features to enable SQL Server change data capture . But it did not address the issue fully and it was only in 2008 that Microsoft SQL Server launched the Change Data Capture technology where developers could capture and archive data without additional programming. CDC or Change Data Capture is the process that tracks all changes made in user-created tables. These changes are stored in relational tables for quick retrieval and access later with the T-SQL. After the features of the CDC are appl...

Make Amazon Redshift as AWS DMS Target Database

Image
  Amazon Redshift database is a data warehouse solution based in the cloud. With Amazon Database Migration Service, you can migrate data from supported databases keeping Redshift as the target. However, it is essential to ensure that the AWS account and the AWS region form a part of the same Redshift cluster. The first step in DMS REDSHIFT database migration is to move data to an Amazon S3 bucket with AWS DMS. Once the files are located in the S3 bucket, the Amazon S3 automatically transfers the intended tables to the Amazon Redshift data warehouse. The S3 bucket, as pointed out previously has to be in the same AWS Region as the Amazon Redshift database. However, when AWS CLI or DMS API is used for DMS Redshift migration, an AWS Identity and Access Management (IAM) path have to be set up to get access to S3. Amazon Redshift provides automated processes in the following cases. ·         Modifications made at source tables by incremental dat...