AEM Replication queue monitoring using AWS CloudWatch

This post will explain the approach to monitor the Replication Queues of Adobe Experience Manager (AEM) using AWS CloudWatch logs.

Issue:

1. The AEM content replication appears to be blocked.
2. Editors can create content, but the activated pages on the CQ5 publish instance are not updated.

Replication issues will occur with the following scenarios:

1. The publisher instance is not accessible
2. Author to Publisher connectivity is missing
3. Bulk activation of pages or DAM assets
4. Authentication and authorization issue due to credentials mismatch.
5. DNS misconfigurations in the Transport section.
6. Transport users not having to write permission for the replicated content on the target instance
7. Conflicts when writing content to the oak repository (MongoDB repository)
8. Network issues

In case there isn’t any monitoring in place, usually, the customer notifies the AEM Support team whenever there is an activation issue or the activated contents are not updated in the publisher instance. We recommend ensuring that replication queue monitoring is enabled because Ops Team/Support team will be notified if a page or DAM asset becomes stuck in the replication queue.

In this blog, we will explain one of the methods for monitoring the replication queue –  CloudWatch logs. Using this monitoring method, the appropriate team will be notified if there are any issues in the replication queue. 

Here are the 5 steps to implement the replication Queue Monitoring using Cloudwatch:

1. Create a replication logger in the AEM instance.
2. Install and Configure the Cloudwatch logger in the AEM instance.
3. Create an SNS Topic.
4. Setup the Cloudwatch replication logger.
5. Configure Notification details in the AWS SNS. 

Step:1  Create a replication logger in the AEM instance

Navigate to OSGI console -> Configuration tab

1. Search logger for Apache Sling Logging Logger Configuration.
2. Name of the Log file: replication.log
3. Message pattern: same
4. Logger: com.day.cq.replication
5. Save the configuration.

Step:2  Install and Configure Cloudwatch logger in AEM instance

1. Login to the AEM author instance for which we want to set up cloudWatch monitoring.

2. Run the following commands:

sudo yum install awslogs

3. Configure the awslogs.conf file to pass the logs to cloudwatch:

vim /etc/awslogs/awslogs.conf

[/mnt/aem/author/crx-quickstart/logs/replication.log]

datetime_format = %b %d %H:%M:%S

file = /mnt/aem/author/crx-quickstart/logs/replication.log

buffer_duration = 5000

log_stream_name = {instance_id}

initial_position = start_of_file

log_group_name = Replication-log

4. Start and enable awslogsd service by typing the command:

                  sudo service awslogsd start

                  sudo systemctl  enable awslogsd

Step 3: Create an SNS Topic

1. Login to AWS Console
2. Search for SNS service
3. Navigate to SNS > Topic > Create Topic.
4. Select the Standard option and then Give your Topic name

• Now leave the rest of the options blank (Fill if required)
• Create the Topic
• Now we can add Subscriptions (Subjects whom we want to notify)
• Navigate to SNS > Subscriptions > Create Subscription
• Now Create Subscription 
• Select a Topic (Which we created earlier)
• Select Protocol (Email in our case)
• Select Endpoint (Email address in our case)
• Leave the rest of the options blank (fill in if necessary)
• Create a subscription.
• We can add multiple subscriptions to a single topic.

Step: 4: Setup the Cloudwatch replication logger

Specify metric and conditions

• Navigate  to Cloudwatch service -> log groups > log streams > metric filters
• Create metric filter
• Set the filter as “Failed” (if any Failed entry comes in the newly created replication log – SNS will trigger an alert)
• If we can keep the filter pattern as “401” – we will be notified if there is any unauthorized error in the replication log.
• Select your log stream. 
• Test the pattern
• Click on next
• Give the metric name
• Select Metric Namespace
• Give Metric Value (1 in our case)

Leave the rest and click on next.

• Review and create and Metric
• Select the metric filter (checkbox in top-right corner) and click on “create the alarm
• Select the Metric name
• Select the Statistics as “Sample Count”
• Select Period accordingly (default : 5)
• Select Threshold type “Static”’
• Now select the condition Whenever <Metric Name> is “Greater/Equal” 
• Than : <your value>

Select the additional configuration accordingly (Actions on Insufficient data)and the number of data points to alarm.

Step 5: Configure Notification details in the AWS SNS

1. Navigate to AWS SNS Service
2. Define the alarm state trigger in the AWS SNS.
3. IN ALARM > Select SNS Topic 
4. Either select an existing topic Or create a topic
5. Create another alarm for recovery alert when the replication queue is cleared. 
6. Add name and description 
7. Preview and create Alarm.

Add the following commands to your /.bashrc file to automatically reload the cloudwatch log; otherwise, on restarting the AEM instance, awslog will stop updating:

sudo echo "Starting awslogsd"

sudo service awslogsd restart

sudo systemctl enable awslogsd

sudo systemctl status awslogsd