Breaking the Barriers: Visualise Namespaces across MongoDB Atlas Clusters

You have 3 microservices, customers, orders and payments each uses its separate MongoDB Atlas cluster. These clusters may also be in separate Atlas projects. Each cluster has its own set of Collections. Overtime, these set of “collections” or “namespaces” grow in number and eventually becomes difficult to keep track of.

Microservices with individual clusters

This is also true if you are running a multi-tenant SaaS application where customers are spread across multiple such clusters.

While there are multiple ways to keep track of these namespaces, in this article, we will use Atlas Functions from Atlas App Services to keep track of all the collections across multiple clusters. We will then create a dashboard for this using Atlas Charts for easier tracking.

Demo:

• Create an Atlas App Services Application.
• Link two data sources i.e. two Atlas clusters.
• Create a MongoDB collection ( bizviz.nsinfo ) in a separate cluster to hold the namespaces data. You can even use a free tier cluster of MongoDB for this.
• Enable Atlas Data API and create a Data API Key for accessing it. (We are using Data API for insertion here as this will allow us to use the same code in other Atlas Projects and be able to bring the namespaces data into a centralised location.)

Data API

• Create an Atlas Function with the self-explanatory code snippet below (use System authentication).

// args is an array of service names
// args = ["mongodb-atlas","bizprod"]

exports = async function getInfo(request) {
  
  const axios = require('axios').default;
  
  // We are calling this function through an HTTP endpoint  
  const serialized = request.body.text();
  const args = JSON.parse(serialized)["k"];
  
  //List of sources
  let sources = []
  for(let i in args) {
    sources.push(context.services.get(args[i]))
  }

  //Admin databases which will allow querying for all databases in the sources
  let sourceAdmins = []
  for (let i in sources) {
    sourceAdmins.push(sources[i].admin())
  }
  
  //Get all databases and collections in a list
  let listOfNamespaces = []
  
  for(let s in sourceAdmins) {
    let dbs = sourceAdmins[s].getDBNames();
    for (let d in dbs){
      let colls = sources[s].db(dbs[d]).getCollectionNames()
      // console.log("DATABASE: ", dbs[d])
      for (let c in colls) {
        var ns = {}
        // console.log(colls[c])
        // console.log("sources", JSON.stringify(sources[s]))
        ns["date"] = new Date()
        ns["cluster"] = args[s]
        ns["db"] = dbs[d];
        ns["coll"] = colls[c];
        
        listOfNamespaces.push(ns)
      }
      }
    }
  
  // URL of the Atlas Data API that we are using to push the data to a MongoDB collection
  let url = await context.values.get("nsInfoDBEndpoint")
  
  // Key to use for calling the Data API
  let apikey = await context.values.get("data-api-key")
  
  // Prepare the Data API call with all fields
  let data = {
    "url": url,
    "method": "post",
    "data": {
    "collection":"nsinfo",
    "database":"bizviz",
    "dataSource":"CSPersistentStore",
    "documents": listOfNamespaces
    },
    "headers":{"Content-Type":"application/json", "Access-Control-Request-Headers": "*", "api-key": apikey}
  }
  
  // Call the Data API
  const res = axios(data).then((x) => {console.log(x)})
  
  return res
  }

• In the above code, we are also utilising context.values for clean and secure access to the Data API URL and Data API Key.

context.values

• Create an HTTPS endpoint which calls this function.

HTTPS endpoint

Postman: Testing API access to Custom HTTPS endpoint

• On successful execution, the namespaces data will be recorded in nsinfo collection.

Namespace documents

• Create an Atlas Charts Dashboard with nsinfo collection as the data source. Create a few charts using this data. e.g: We want to see the distribution of our collections across the clusters and the list of all collections. Below is the Charts dashboard showing this info.

Charts Dashboard showing Namespaces across clusters

Some points worth noting:

• Extensibility: In this demo, we have picked only 1 project with 2 clusters. You can create a copy of the same App Services app into other Atlas projects and links its clusters. The Data API for inserting the data into the collection will remain the same. In case you plan to use across multiple projects, I suggest collecting another field in the document with project ID/name for easier reference to the clusters in your MongoDB landscape.
• Automation: We are using custom HTTPS endpoint to collect the data which needs to be run manually. We can instead use Scheduled Triggers to collect this automatically say every morning and use TTL indexes on our nsinfo collection to remove the older data. Effectively, you’ll refresh this dashboard on a daily basis.
• Scaling: In some environments, there may be multiple Atlas clusters in a single project and some clusters may have large number of databases and collections. In this case, the code to get all the namespaces may timeout. One strategy to fix this is to break the function down into multiple functions to first get the list of clusters, then get the databases, and then get all the collections in each database. This will need writing the data at each intermediate step and then processing the clusters/databases separately.

I hope this article was useful. Let me know your thoughts in the comments and don’t forget to send a clap if you enjoyed reading this.

Cheers!