Manager Worker Pattern for Atlas App Services
In the last two articles, we explored how we can consolidate the namespaces from all the clusters in a project in a single dashboard, and separately also explored how we can monitor the free space in those collections. At the end of the last article I promised to combine the learnings of these two pieces and add further on it to build a wholistic dashboard to monitor the namespaces organisation wide.
In the spirit of agile development, this article will share a part of that promise while also providing another tangible building block that you can use in your other applications.
In MongoDB Atlas, each App Services app can access the clusters in its own project. There may be a situation when the data is distributed across many clusters spread across multiple projects. We solved this exact problem in the first article in this series by having the projects send data over to a central project via Data API.
This was ok for a few projects but as the number of projects increase sometimes tens to hundreds, this can lead to the problem of managing and deploying multiple apps. There needs to be an automated way to solve this issue.
Manager Worker Pattern in App Services
Manager Worker Design Pattern is common in software development. We extend it here to make it work with App Services. In effect, we will deploy a single Manager App which will be responsible for deploying and managing the Worker Apps. The Worker Apps (like agents) will collect the data from their projects and publish it to the Manager App (or its database).
Manager App Architecture
Manager app will contain two major components: (1) User Interface (UI), and (2) Backend.
We are using a NextJS app hosted on Vercel to build the UI. The UI is simple at the moment with only 3 textboxes which are pretty much self-explanatory.
The UI also makes use of Realm Web SDK to call the functions from the backend application.
The backend component is an App Services application which does the following:
- Gets the project keys from the UI
- Stores the project keys in the Backing DB
- Gets the cluster information from the project using the keys and stores it in the Backing DB
- Deploys the Worker App in the project using the App Services API plus the keys provided by the user in step #1
- After deploying the worker app, the Backend links it to all the clusters in its project and provides it the necessary functions that it needs to call to collect the data
- Worker App calls these functions which collects the monitoring data from its clusters and sends that data to the Manager App ( at the moment writing directly to its Backing DB using Data API)
- UI will be extended further to visualise the data stored in the Backing DB of the Manager App
Introducing GlobeTrotter
When I kickstarted the repository to bring this architecture to life, the first thing was to choose the right name. As I imagined the data traveling from multiple Atlas projects into a single place, it felt like a person taking a journey across the world. Hence the name GlobeTrotter.
Stay tuned for updates on this repo!
Don’t forget to send a clap if you liked this article.
