This blog explains task coordination in a clustered WSO2 Micro Integrator environment. When scheduled tasks are deployed across multiple nodes, they may run simultaneously without proper coordination, causing duplicate message processing and inconsistencies. WSO2 MI solves this using an RDBMS-based coordination mechanism through the WSO2_COORDINATION_DB.
The blog outlines the required database setup, deployment configuration, and explains available task resolvers: Default, Round Robin, and Task Node Resolver. With correct configuration, tasks run on only one node at a time. If a node fails, the task is automatically reassigned, ensuring high availability, fault tolerance, and reliable clustered operations.
Task coordination in a clustered setup is the process of managing and orchestrating the execution of tasks across multiple nodes within a cluster. In such a setup, task coordination ensures that the tasks are distributed, executed, and managed in an efficient manner, avoiding conflicts, redundancy, and resource contention. Simply put, this process ensures that tasks are executed efficiently, avoiding conflicts and maintaining proper synchronization between different nodes.
WSO2 MI is an integration runtime. In a WSO2 MI clustered setup, ensuring that tasks are coordinated correctly across multiple instances of MI is crucial to avoid redundancy and ensure consistency.
In a clustered setup, the same scheduled task might be deployed on multiple nodes. Without proper coordination, the task may be executed multiple times on each node, leading to issues like duplicate processing, which can eventually result in data corruption or performance degradation.
In WSO2 MI most of the integration artifacts in the deployment are stateless and do not require coordination when deployed in a clustered setup. But the following artifacts require coordination when deployed in multiple instances,
- Scheduled Tasks
- Message Processors
- Polling Inbound Endpoints
- Event-Based Inbound Endpoints
Create a Simple Task
The scenario is to deploy a sample scheduled task to run every 30 seconds in two servers/instances of WSO2 MI locally. For this purpose, I’m using the latest WSO2 Micro Integrator 4.3. You can refer to one of our blogs on how to run WSO2 products on Linux/Windows/Mac.
As mentioned, I’ll be running two servers of WSO2 MI 4.3 locally by running on two different ports. One server would be running on the default port and the other server will be running by modifying the port offset value. The port offset will remain at the default setting for one of the servers.
On the other server the port offset can be updated as below by updating the deployment.toml file in <wso2mi-home>/conf,
Create an integration project and develop a simple task as below along with a sequence to inject when the task gets executed.
This is a simple configuration where the task will be executed every 30 seconds, once the task gets executed, it will call the sequence configured in injectTo parameter. Once the sequence is called, the sequence will log the message.
Deploy and Test before Task Coordination Configuration
Build the integration project using Maven and copy the exported carbon application to the < wso2mi-home> /repository/deployment/server/carbonapps folder of both the servers and start both the integration servers.
As shown below, the task has been deployed, scheduled, and is running on both servers.
Ideally, the task should be running only in one server in a clustered setup. If the task runs on both the servers, there will be duplication of messages or processes.
Task Coordination Configuration
Task coordination between nodes in a clustered setup in WSO2 MI is achieved with RDBMS. In simple, for task coordination the nodes should communicate, and this communication happens via a database.
The communication between the nodes in WSO2 MI happens via the WSO2_COORDINATION_DB. The script to create the database is in <wso2mi-home>/dbscripts/<db_type>. To create the database, run the <db_type>_cluster.sql script in <wso2mi-home>/dbscripts/<db_type> (eg: if the DB type is MySQL then run the <wso2mi-home>/dbscripts/mysql/mysql_cluster.sql).
For the communication of WSO2 MI server and the created database add the following snippet to deployment.toml under <wso2mi-home>/conf. I have used MySQL hence the below config is to connect to the created MySQL database.
Add the relevant database driver to the <wso2mi-home>/lib directory.
Deploy and Test after Task Coordination Configuration
Already a simple task is deployed in the two WSO2 MI servers. Once the required task coordination configurations are done, start both the WSO2 MI servers. Once the servers are started, it is clear that the task has got deployed on both servers but scheduled and running only on one server.
Even though the task is deployed on both the servers the task has got scheduled and is running only on one server as expected. As the task is running only in one instance, there will be no duplication of messages. Not just duplication, but also availability and fault tolerance can be achieved. This can be tested by taking down the server which has scheduled the task.
As per the above image, the server which scheduled the task is taken down, but the task got scheduled to the next available server. This proves that tasks are fault tolerant.
Task Resolvers
Task Resolver plays a role in ensuring that tasks are correctly coordinated across multiple nodes. It may determine which node should execute a particular task, thereby preventing duplicate executions and ensuring that tasks are distributed effectively. When there are tasks deployed in WSO2 MI, each task should run only in one node of the cluster. The task resolver configuration in the server nodes specifies the logic for allocating tasks to the server nodes.
Default Resolver
This is default task resolver of WSO2 MI and this does not need any resolver configuration. The default task resolver works in an active passive mode where all the tasks are resolved to a random node from the available nodes in the cluster. In case the server running the tasks becomes unavailable, the tasks will be resolved to some other node.
Round Robin Resolver
This resolver distributes the tasks among the nodes in a circular order. With this resolver, tasks can be evenly distributed among the nodes in the cluster, rather than all tasks running on a single server. Also, by distributing the tasks in a round robin fashion, the system achieves better load balancing. Following configuration should be added in deployment.toml under <wso2mi-home>/conf,
For the round robin resolver, the task_server_count parameter is the number of nodes required to be in the cluster before starting the task resolving process. Hence, this ensures that all the tasks are not scheduled on a single server until the required number of servers are available to distribute the tasks in a round robin manner.
Task node resolver
This resolver also distributes the tasks among the nodes in a round robin manner but to a predefined set of nodes. Instead of resolving the task in all available nodes, tasks can be resolved among the required nodes. Following configuration should be added in deployment.toml under <wso2mi-home>/conf,
Above is a config in one of the nodes and has the ID node-1. Likewise, all the nodes in the cluster are required to be provided with an ID so that the required set of nodes which should resolve the tasks can be set in the task_nodes parameter. As per the above config, the tasks will be resolved in node-1 and node-2 only.
Additionally, for the other two node resolvers, a node ID was not given to the nodes. If needed, IDs can be provided to other node resolvers too, but not required. If an ID is not provided a random UUID value will be used as the node ID. But for the task node resolver, ID should be provided for nodes so that task_nodes parameter can be filled in with the IDs.
Conclusion
Effective task coordination ensures that a clustered setup operates efficiently, minimizing downtime and maximizing resource utilization. It helps in scaling applications, achieving high availability, and managing complex workloads that span multiple nodes.
Task coordination in WSO2 Micro Integrator is crucial for ensuring that tasks are executed efficiently and without redundancy in a clustered environment. WSO2 MI also gives the flexibility to choose a node resolver based on the requirements.
With the required coordination configuration in WSO2 MI, duplication of messages or duplication of integration mediation are handled efficiently preventing redundant operations. Further, ensures that tasks can be reassigned to other nodes if a failure occurs. This enhances the system’s fault tolerance and ensures continuous operation. Hence this is an essential feature for any large-scale, clustered deployment of WSO2 Micro Integrator.
