When developing services in WSO2 Micro Integrator, it’s often essential to add retry mechanisms for reliable communication with back-end systems that may be temporarily unavailable. Exponential backoff is one of the approaches to manage retries, where the delay between each retry attempt progressively increases.
This blog post explores how to configure exponential retries in WSO2 MI, using an example with ActiveMQ to showcase how to handle message redelivery on failure.
I have tested this setup locally on WSO2 MI 4.2 (U2 level – 102) with ActiveMQ 6.0.1 as a message broker.
Overview of Exponential Backoff in WSO2 Micro Integrator
Exponential backoff is a retry strategy where the wait time between retries increases exponentially. The strategy helps in situations where the backend server might experience temporary issues or be overloaded. With exponential backoff, you start with a base retry interval and subsequent delays increase based on a backoff multiplier until a specified maximum delay. This method minimizes unnecessary load and optimizes recovery time for both the client and the server.
Let’s start by configuring the retry mechanism in the proxy service code sample.
Configuring JMS Proxy with Exponential Backoff
In this example, we set up a consumer proxy service that listens to a JMS queue and makes an HTTP call to a backend endpoint. If the backend call fails, the JMS message should be redelivered with exponential backoff intervals.
Key Parameters for Exponential Backoff
In WSO2 MI, we can configure exponential backoff retries by setting parameters in the JMS proxy service
redeliveryPolicy.useExponentialBackOff : Enables exponential backoff for retries.
redeliveryPolicy.backOffMultiplier : Sets the factor by which the delay is multiplied after each retry.
redeliveryPolicy.initialRedeliveryDelay : Sets the delay before the first retry.
Sample JMS Consumer Proxy Configuration
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="ExponentialRetryProxySample" startOnLoad="true" transports="jms" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<log level="custom">
<property name="MSG" value="Receiving messages from Queue"/>
<property expression="json-eval($)" name="Payload"/>
</log>
<call blocking="true">
<endpoint>
<http method="GET" uri-template="http://backend.example.com/api/resource">
<suspendOnFailure>
<errorCodes>-1</errorCodes>
<initialDuration>-1</initialDuration>
<progressionFactor>1</progressionFactor>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</http>
</endpoint>
</call>
<drop/>
</inSequence>
<outSequence/>
<faultSequence>
<log level="custom">
<property name="MSG" value="Fault Sequence Triggered"/>
</log>
<property name="SET_ROLLBACK_ONLY" scope="axis2" value="true"/>
</faultSequence>
</target>
<parameter name="transport.jms.DestinationType">queue</parameter>
<parameter name="transport.jms.Destination">testqueue</parameter>
<parameter name="transport.jms.ContentType">
<rules>
<jmsProperty>contentType</jmsProperty>
<default>application/json</default>
</rules>
</parameter>
<parameter name="transport.jms.ConnectionFactory">myQueueListener</parameter>
<parameter name="transport.jms.SessionAcknowledgement">CLIENT_ACKNOWLEDGE</parameter>
<parameter name="transport.jms.SessionTransacted">true</parameter>
<parameter name="transport.jms.CacheLevel">consumer</parameter>
<parameter name="redeliveryPolicy.backOffMultiplier">3</parameter>
<parameter name="redeliveryPolicy.maximumRedeliveries">5</parameter>
<parameter name="redeliveryPolicy.initialRedeliveryDelay">2000</parameter>
<parameter name="redeliveryPolicy.useExponentialBackOff">true</parameter>
</proxy> Explanation of Key Elements
- endpoint Configuration:
– suspendOnFailure: Configures the endpoint to avoid suspension on failure, so retries continue based on the JMS redelivery policy.
– markForSuspension: Prevents the endpoint from entering a suspended state after failure. - faultSequence: Executes when the endpoint fails to respond. The SET_ROLLBACK_ONLY property, set to “true”, marks the transaction for rollback, triggering message redelivery with exponential backoff.
- JMS Parameters:
– redeliveryPolicy.useExponentialBackOff: Enables exponential backoff.
– redeliveryPolicy.backOffMultiplier: Sets the backoff multiplier to 3, tripling the delay after each retry.
– redeliveryPolicy.initialRedeliveryDelay: Sets the initial delay before the first retry to 2000 ms (2 seconds).
– redeliveryPolicy.maximumRedeliveries: Limits the retries to five attempts.
How Exponential Retry Works
With the above settings, retries occur at exponential intervals:
- First retry: 2 seconds
- Second retry: 6 seconds (2s * 3)
- Third retry: 18 seconds (6s * 3)
- Fourth retry: 54 seconds (18s * 3)
- And so on, up to a maximum of five retries.
If a message fails after all retry attempts, it is sent to a Dead Letter Queue (DLQ) for manual inspection.
Additional Configuration Tips
- Blocking Mode: Setting blocking=”true” in the call mediator ensures that the system waits for the HTTP response. This approach is essential for reliable redelivery in JMS proxies.
- Preventing Endpoint Suspension: By default, when an endpoint failure is detected, the endpoint will be suspended for 30,000 milliseconds. When an endpoint is in suspended state for a specified time duration following a failure, it cannot process any new messages but, in this case, the suspendOnFailure and markForSuspension configurations help keep the endpoint active for subsequent retries.
- Fine-Tuning Redelivery Policy: You can further adjust the redelivery behavior using additional parameters:
a. redeliveryPolicy.maximumRedeliveryDelay: Defines the cap in seconds, ensuring retry intervals don’t grow excessively long.
b. redeliveryPolicy.maximumRedeliveries: Sets the cap on the number of retries. - Message Transaction Control: Setting transport.jms.SessionTransacted to “true” ensures that the message is only removed from the queue once it has been successfully processed.
Configuration in deployment.toml
[[transport.jms.sender]]
name = "myQueueSender"
parameter.initial_naming_factory = "org.apache.activemq.jndi.ActiveMQInitialContextFactory"
parameter.provider_url = "tcp://${activemqEndpoint}" //input your activemq URL
parameter.connection_factory_name = "QueueConnectionFactory"
parameter.connection_factory_type = "queue"
parameter.cache_level = "producer"
[[transport.jms.listener]]
name = "myQueueListener"
parameter.initial_naming_factory = "org.apache.activemq.jndi.ActiveMQInitialContextFactory"
parameter.provider_url = "failover:(tcp://${activemqEndpoint})" //input your activemq URL
parameter.connection_factory_name = "QueueConnectionFactory"
parameter.connection_factory_type = "queue"
parameter.cache_level = "consumer"
[[transport.jms.listener]]
name = "default"
parameter.initial_naming_factory = "org.apache.activemq.jndi.ActiveMQInitialContextFactory"
parameter.provider_url = "failover:(tcp://${activemqEndpoint})"
parameter.connection_factory_name = "QueueConnectionFactory"
parameter.connection_factory_type = "queue" After adding these configurations, you need to restart MI.
Testing and Observing Exponential Backoff
To initiate the test, we published a sample message to the testqueue through the ActiveMQ Web Console (http://localhost:8161/admin). The queue and connection factory were preconfigured in ActiveMQ, and the WSO2 MI JMS listener was subscribed to this queue via the ExponentialRetryProxySample proxy.
When the proxy received a message from the queue, it attempted to invoke the configured. To simulate endpoint failure, we can use an invalid endpoint URL.
This forced the faultSequence to trigger, marking the transaction for rollback using the property.
<property name="SET_ROLLBACK_ONLY" scope="axis2" type="STRING" value="true"/> As a result, ActiveMQ retried message delivery according to the configured exponential backoff policy. We observed that retry intervals increased progressively from 2 seconds → 6 seconds → 18 seconds, and so on, based on the backOffMultiplier value of 3.
Once the maximum retry limit (defined by redeliveryPolicy.maximumRedeliveries) was reached, the message was automatically moved to the Dead Letter Queue (DLQ) for further analysis.
Sample Log Output
Here’s an example of the log output during testing, showing retry intervals of 2 seconds, 6 seconds, and 18 seconds and so on:
[2024-11-05 14:07:33,734] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Receiving messages from Queue, Payload = {
"test": "message"
}
[2024-11-05 14:08:33,768] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Fault Sequence Triggered
[2024-11-05 14:08:33,769] ERROR {ServiceTaskManager} - Error recovering the JMS session javax.jms.IllegalStateException: This session is transacted
[2024-11-05 14:08:35,807] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Receiving messages from Queue, Payload = {
"test": "message"
}
[2024-11-05 14:08:35,808] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Fault Sequence Triggered
[2024-11-05 14:08:35,809] ERROR {ServiceTaskManager} - Error recovering the JMS session javax.jms.IllegalStateException: This session is transacted
[2024-11-05 14:08:41,824] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Receiving messages from Queue, Payload = {
"test": "message"
}
[2024-11-05 14:08:41,825] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Fault Sequence Triggered
[2024-11-05 14:08:41,826] ERROR {ServiceTaskManager} - Error recovering the JMS session javax.jms.IllegalStateException: This session is transacted
[2024-11-05 14:08:59,843] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Receiving messages from Queue, Payload = {
"test": "message"
}
[2024-11-05 14:08:59,844] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Fault Sequence Triggered
[2024-11-05 14:08:59,844] ERROR {ServiceTaskManager} - Error recovering the JMS session javax.jms.IllegalStateException: This session is transacted
[2024-11-05 14:09:53,864] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Receiving messages from Queue, Payload = {
"test": "message"
}
[2024-11-05 14:10:53,875] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Fault Sequence Triggered
[2024-11-05 14:10:53,875] ERROR {ServiceTaskManager} - Error recovering the JMS session javax.jms.IllegalStateException: This session is transacted
[2024-11-05 14:13:35,900] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Receiving messages from Queue, Payload = {
"test": "message"
}
[2024-11-05 14:14:35,921] INFO {LogMediator} - {proxy: ExponentialRetryProxySample} MSG = Fault Sequence Triggered
[2024-11-05 14:14:35,921] ERROR {ServiceTaskManager} - Error recovering the JMS session javax.jms.IllegalStateException: This session is transacted This log output confirms that retries follow exponential intervals as configured.
Conclusion
Exponential retry in WSO2 Micro Integrator is an effective way to handle temporary backend failures while maintaining reliable JMS message processing. By using transactional sessions together with an exponential backoff redelivery policy, you can reduce unnecessary load on failing services and improve overall system resilience.
Try this configuration in your own WSO2 MI environment and experiment with the redelivery parameters to find the optimal balance for your use case. If you have insights or alternative approaches, feel free to share them by reaching out to one of our account managers.
