Listing Thumbnail

    Amazon MSK Connect MM2 Plugin

     Info
    AWS Free Tier
    Amazon MSK Connect MM2 Plugin
    Listing Thumbnail

    Amazon MSK Connect MM2 Plugin

     Info

    Overview

    A solution based on the Apache Kafka MirrorMaker 2.0 (MM2) replication engine, that satisfies the important use-case of maintaining real-time replication between:

    1. Self-managed Kafka environment on-premises and the Amazon MSK cluster (provisioned or serverless)
    2. Self-managed Kafka environment on AWS Cloud and Amazon MSK cluster (provisioned or serverless)
    3. Amazon MSK cluster and Amazon MSK cluster (provisioned or serverless)

    The solution for Amazon MSK Connect, which includes native integration to services such as Amazon CloudWatch and Amazon Simple Notification Service (SNS).

    Highlights

    • Ease of Setup
    • Built on MSK Connect platform
    • Enhanced replication monitoring & alerting

    Details

    Delivery method

    Delivery option
    MSKConnect-Connectors-Replication-Plugin

    Latest version

    Operating system
    AmazonLinux Amazon Linux 2023 (AL2023)

    Pricing

    Amazon MSK Connect MM2 Plugin

     Info
    Pricing is based on contract duration. You pay upfront or in installments according to your contract terms with the vendor. This entitles you to a specified quantity of use for the contract duration. If you choose not to renew or replace your contract before it ends, access to these entitlements will expire.
    Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator  to estimate your infrastructure costs.

    1-month contract (1)

     Info
    Dimension
    Description
    Cost/month
    Standard Contract
    Standard use of Conerstone's MM2 plugin
    $100.00

    Vendor refund policy

    Contact sales@cornerstone-consulting.io  for refund requests.

    Legal

    Vendor terms and conditions

    Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA) .

    Content disclaimer

    Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.

    Usage information

     Info

    Delivery details

    MSKConnect-Connectors-Replication-Plugin

    Within the EC2 instance you launched, with the CloudFormation template, is a folder named “Amazon-MSK-Connect-MM2-replication-plugin-by-CCG-1.0” that holds: Two CloudFormation templates “MSKConnect-Connectors-Monitoring_CF.json” and “MSKConnect- Connectors-Monitoring-DestinationMSKClusterIAMAuthentication_CF.json” Plugin binary zip file “cc-mm2.zip” User guide.

    Plugin setup is comprised of 3 main steps: MSK Connect Worker configuration creation Upload of the plugin binary file to the Amazon MSK Customised plugins Deployment of the Cloud Formation template that corresponds to your destination MSK Cluster configuration

    CloudFormation template creates: Amazon MSK Connect Connectors CloudWatch log groups CloudWatch metric filters CloudWatch Alarms and the SNS topic IAM role used by Amazon MSK Connectors (Source Connector, Checkpoint Connector, and Heartbeat Connector)

    The plugin binary zip file will be uploaded to one of your S3 buckets through CLI of your EC2. Amazon MSK Worker configuration and the Amazon MSK Customised plugin will be created via AWS console. Lastly, the CloudFormation stack will be deployed either through CLI command or through AWS console which will require the upload of the template to S3 bucket prior to the CloudFormation stack creation.

    Once the infrastructure is set, the data replication between the source and the destination cluster will immediately start taking place as long as both source and the destination clusters are up and running. Monitoring will view the logs created by the connectors and send an alarm notification through the SNS topic.

    To receive alert notifications, you must subscribe to the SNS topic once the CloudFormation stack creation is completed. Alerts will be sent for the following scenarios: Data replication is not taking place between the clusters Clusters are not responding

    CloudFormation Template (CFT)

    AWS CloudFormation templates are JSON or YAML-formatted text files that simplify provisioning and management on AWS. The templates describe the service or application architecture you want to deploy, and AWS CloudFormation uses those templates to provision and configure the required services (such as Amazon EC2 instances or Amazon RDS DB instances). The deployed application and associated resources are called a "stack."

    Version release notes

    First Release

    Additional details

    Usage instructions

    See user guide: https://cornerstone-offerings-public.s3.amazonaws.com/msk-mm2-plugin/Cornerstone_MSK_MM2_Plugin_1.0_SetupGuide.pdf 

    • Amazon Marketplace will lead you to the deployment of EC2 instance through CloudFormation stack in the region you choose. EC2 will come with a profile given the least privilege necessary to deploy the replication plugin. • You will be asked to provide a Subnet ID, Security Group ID, and the name of the key you will be using to SSH into your instance.
    • Once the stack creation is completed navigate to your Amazon EC2 console and verify that you're in the region you have chosen when spinning up the instance. • Choose the Instance you just launched and connect to it. • Navigate to the /home/ec2-user/Amazon-MSK-Connect-MM2-replication-plugin-by-CCG-1.0 directory. • Confirm that the "ccg-mm2", "MSKConnect-Connectors-Monitoring-DestinationMSKClusterIAMAuthentication_CF.json", "MSKConnect-Connectors-Monitoring_CF.json", and the "Cornerstone_MSK_MM2_Plugin_1.0_SetupGuide.pdf" are there. • Download and follow the Cornerstone_MSK_MM2_Plugin_1.0_SetupGuide.pdf setup guide. • If the CloudFormation stacks fail to deploy, proof check the values of the parameters you provided. Alternatively check the IAM permissions of the EC2 role you are deploying the stack with and add any further permissions necessary. • The EC2 in use is a temporary means for the deployment of the plugin to your AWS Account. Once the infrastructure is set up and the data replication is confirmed, you can clean up the EC2 resource. • User guide contains detailed instruction on how to test and validates the plugin once the setup has completed. • If you have set up the plugin but are not able to observe the data replication happening, or you are not able to connect to the clusters, you can refer to the CloudWatch Alarms and CloudWatch log groups that come with plugin deployment. • The plugin comes with 5 different CloudWatch alarm metrics and 5 CloudWatch alarms which monitor the Kafka/Amazon MSK source cluster health, Destination Amazon MSK cluster health and the occurrence of the data replication between the cluster. The alarm names and observations are:

    • source-cluster-health-timeoutexception - source cluster gives a timeout exception more frequently than is to be considered normal
    • source-cluster-health-DNSnotresolved - source cluster’s DNS address cannot be resolved
    • destination-cluster-health-disconnected - destination cluster is disconnected
    • worker-health-noheartbeat - if worker/s is/are not sending heartbeat logs they are not replicating data
    • source-cluster-health-brokerunavailable - source cluster is not available

    • In order to receive the alarm notifications, subscribe to the "msk-mm2-health" or "mskIAM-mm2-health" SNS topic, dependent on the CloudFormation template you have created the resources with. • If you see any of the alarms in the “Alarm” state, firstly check the availability of your clusters. If the issue persists you should check the logs of the connectors in the CloudWatch Log groups which will have names either starting with "msk/mm2_Source_Checkpoint_Connector_" or "msk/mm2_Heartbeat_Connector_" and take action corresponding to the logged errors. • If your clusters are up and running, cluster configurations are compatible, and the Kafka version you are using is supported by the Amazon MSK Connect you should not be getting any errors.

    NOTE: No sensitive information saved by customers No encryption configuration No requirement for rotating programmatic system credentials and cryptographic keys No requirement to decrypt data

    Support

    Vendor support

    Contact support@cornerstone-consulting.io  for support queries.

    AWS infrastructure support

    AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.

    Similar products

    Customer reviews

    Ratings and reviews

     Info
    0 ratings
    5 star
    4 star
    3 star
    2 star
    1 star
    0%
    0%
    0%
    0%
    0%
    0 AWS reviews
    No customer reviews yet
    Be the first to write a review for this product.