Feeding MongoDB Atlas Alerts to AWS CloudWatch

MongoDB Atlas is the cloud database service from MongoDB. It allows customers to run MongoDB seamlessly across all three major cloud providers i.e. AWS, GCP and Azure.

Atlas offers strong monitoring and alerting capabilities. These alerts can be fed to systems like PagerDuty, Slack and Datadog with out of the box integrations. However, in some complex environments, there may be a need to send these alerts into some other third party systems for further actions. For this, Atlas allows you to configure Webhooks to which it can then send a POST request as soon as an alert is received.

In this post, our eventual goal is to forward these alerts to AWS CloudWatch from where they can be consumed by other systems such as ServiceNow. Realm which is part of the broader MongoDB’s Application Data Platform has several capabilities which will help us in this integration.

Below is the high level idea:

1. We will use Realm to host a webhook. This webhook will receive the alerts when one is triggered.
2. On receiving an alert, the webhook will write that alert into a collection on another MongoDB database on Atlas.
3. On this collection, we will create a Realm Trigger which has a direct integration to AWS EventBridge.
4. AWS EventBridge will have the CloudWatch logs attached to it.

High Level Architecture of moving the Atlas Alert to AWS CloudWatch

Walkthrough of the process:

Step 1: Create two Atlas clusters

• “Cluster 0” (Production Cluster using M10 instance hosted in AWS Mumbai Region)
• and “Cluster 1” (Alerts Database using M10 instance hosted in AWS N.Virginia) on Atlas.

Step 2: Create the collections in both clusters

• Cluster 0 has a collection named orders in the testingOA database.
• Cluster 1 has a collection named alert_collection in the alerts database.

Step 3: Create a Realm Application & Configure Rules

• Click Realm tab on top of your Atlas console.

Realm in Atlas Console

• Click “Create a New App” button.
• Provide a name to your Realm app and link it to Cluster 1

Name the application and link it to cluster containing the alerts database

• This will create a default “mongodb-atlas” service in the Linked Data Sources tab on the left side of your Realm app console.

• Click on Rules button on the left side of your Realm application console.
• Add a rule for the alert_collection

Set the rules for alert_collection

• Enable write permissions on the collection and deploy the app

Enable Write Permissions

Step 4: Create the webhook

• Go to 3rd Party Services and select HTTP. Name the service “my-alert-service” and Click Add Service.

Create the HTTP Service

• Click Add Incoming Webhook button.

Add Incoming Webhook

• In the Settings page, name the webhook “receiveAlertsWebhook”. Leave everything else on this as default. It’ll use System authentication and HTTP POST method.

Webhook Settings

• Go to Function Editor. This is the place where we will put the logic to get the alert into our Alerts Database. Add the following code. Save Draft, Click Review and Deploy on the top.

• Copy the Webhook URL from the Settings tab of the Incoming Webhook.

Step 5: Setup the Webhook Integration with Atlas Alerts

• Go to Integrations page of your Atlas project.
• Click Webhook Settings. A dialog box will appear on the screen.
• Paste the Webhook URL in this dialog box, and click Activate.

Step 6: Set the desired alerts

• Add an alert in the Alerts page of the project.
• For this basic walkthrough, we will set the Query Targeting Alert. We will run a synthetic workload to generate this alert to test that our integration is successful.

Configure the alert choosing Webhook in the Send To section

Step 7: Test this setup

• You can run a synthetic workload (frequent queries which will scan all documents of the collection without using an index) to generate this alert. When the alert is generated, the document will be inserted into the collection in Cluster 1. (Alternatively, you can configure a different alert like “Replica Set has elected a new Primary” and use Test Failover button to trigger that)

Alert captured in the collection alert_collection

Step 8: Setup the trigger to forward the event to EventBridge

• The process to setup a trigger and forward the event to EventBridge is detailed here in the MongoDB Realm manual.
• Additionally, on the EventBridge side, click on the Custom Event Bus and Go to Rules and Create Rule.
• Choose a name “myfirstrule” for this > Click Event Pattern > Select Predefined Pattern by the Service > Select All Events.
• In the Targets section, choose Cloudwatch log group, and name it say “eventbridge-alerts”
• This trigger is watching for “insert” operations on alert_collection. When the alert will be logged in the collection, it’ll trigger and forward it to EventBridge.

Step 8: Verify the setup

• Generate the alert again as discussed in Step 7 above.
• Verify from alert_collection, Realm logs and CloudWatch logs that the event has been triggered.

CloudWatch Logs

I hope this blog was useful.

There is probably an alternate approach as well which will not need you to write the alerts into an Atlas cluster first and instead use the AWS SDK directly to forward it to EventBridge. I’ll explore and write about it soon. (I tried this approach, but Realm functions do not currently support using Eventbridge in the SDK. This is expected to come soon.)