The “before” scenario…
Let’s imagine some example scenario’s where a Message Broker solution is not in place. One situation where it is very important to receive real time data updates from is the stock exchange market, where several stock brokers need to react as soon as possible. In this type of market, the sooner you act the better the results. Assume we have a Stock Exchange system with an internal database, where all consumers around have acces to pull data from, we have a situation like in the following picture:
The arrows represent several consumers that need to send requests every time to the central database to get the latest updates on stock rates e.d. Because of that the consumer needs to do a request every time, we have no real time situation. Meaning that the data pulled from the database can be outdated after 1 second again. There is need for a solution to receive real time data from the Stock Exchange Market, without loss of data, in a fast and reliable way.
Other scenario examples that have to deal with the same type of timing issues are electronic bank payments, airline flight departures and arrivals, the health industry,…. For all this a Message Broker solution comes in place! The moment when the Stock Exchange Market has an update on a Stock, a message should be delivered to the message broker, which will handle and deliver this message to the several consumers in a fast and trusted way. A Message Broker like this should be setup as a stable, high available and efficient system.
Introducing the clustered Message Broker
Since WSO2MB version 3.0.0 (Message Broker) it is possible to cluster WSO2MB and let it work with a clustered WSO2ESB environment.
First let’s explain the obvious question: “Why clustering the Message Broker?” Here are the most valid motivations:
- Increased availability of queues, since multiple Message Broker instances are exposed as clustered queues.
- Faster throughput of messages, since messages can be delivered on multiple queues.
- Better distribution of workload based on non-functional requirements.
- Trusted delivery of messages by the Message Broker.
For a long time since the introduction of WSO2MB it was not possible to setup a high available clustered environment together with WSO2ESB that supports JMS send and receive routines with proxies. So we where forced to look into alternative message broker technologies like ActiveMQ that do support some kind of failover / clustering way of working, and integrating it with WSO2ESB.
Since last year with WSO2MB 3.0.0 we have the possibility to configure the message broker with WSO2ESB in a high available way, that leads to a 100% WSO2 solution without incorporating other vendor technologies. In order to realize the setup it is advisable to at least use WSO2ESB 4.8.1, or take the latest ESB version. (now v 5.0.0)
High level overview
In what situations do we need a HA infrastructure setup with JMS? Here are a few common points:
- To interface as a high available system to / from other “legacy” JMS message brokers that already exist in the infra.
- For a more flexible upscaling.
- For a more flexible failover situation.
WSO2 has an excellent online guide regarding clustering of various WSO2 products, which I have included links on reference list. But for clearity I want to explain in main lines how clustering with WSO2 MB and ESB is structured.
First let’s get into MB clustering. MB is able to talk with an external clustered database managed system. (DBMS) Thanks to this ability we have a more stable, trusted message storage persistance for MB. MB supports integration with most common database products like Oracle and MySQL.
This way we ensure a high available (HA) setup with high efficiency. To enable clustering, Hazelcast is used, which will function as the distributor for all the server requests that are sended to the MB cluster, and fluently will manage to distribute over all the various cluster MB instances.
In a same fashion clustering of WSO2ESB is done. Hazelcast will support server request distributions over various nodes with their Hazelcast agents. And message persistance + governance storage can also be done on external DBMS clustered nodes. On top of that various 3rd party loadbalanced solutions are supported like HAProxy or NGINX.
This blog will not get into how to setup a database infrastructure. But it is important for a production-like environment to realize a proper database setup on dedicated environments with optimal support. With optimal support we are talking about various things a DevOps team should be able to have:
- Automated nightly backups.
- Covering of fallback / rollback scenarios.
- Managed failover.
- Fully integrated with automated database changescript deployments.
- Seperation between various environments for development, testing, acceptance testing (UAT) and production.
- Proper technical monitoring etc.
In the end JMS message sender and receiver patterns are covered according to the following pictures.
A request message is received by the HAProxy, and it will redirect the message to ESB node 2. After that ESB node 2 has to send the message to a queue. WSO2ESB connects with the JMS messageSender to the MB cluster. The ESB understands that it has to send the message to MB node 2 as default, with failover to other nodes if MB2 is not available (MB2 → MB3 → MB1), to handle the JMS message. The queue message will be received based on roundrobin failover strategy by MB node 2. From that node the message will be stored in the common message store as a queue message.
Applying dead letter channels in the ESB cluster
During this stage we’ve got our MB and WSO2 clusters setup. Assume we’ve got the situation according to the following picture
The MB and the ESB clusters can run on dedicated virtuals, depending on how much resources are available for your clusters.
Each WSO2ESB worker node should be configured in a way that it can talk with the MB cluster with a failover strategy. In that sence that if for instance MB node 1 crashed, the ESB will handle JMS messages with the next MB node in the brokerlist and so on. Navigate to [ESB_HOME]/repository/conf/jndi.properties and configure the so called “Dead Letter Channel Pattern” in the following way:
Above form is the setup for the JNDI properties of WSO2 ESB worker node 1. It talks to MB node 1 as default. And in a failover situation it switches to MB node 2, and then MB 3. When MB node 1 is recovered, ESB node 1 needs a restart in order to connect to MB node 1 again. But that’s ofcourse no issue, because we have an ESB cluster with maximally 3 available nodes, which ensures high availability.
CAUTION : When dealing with a distributed cluster solution, where the nodes should talk with each other in a remote way (meaning we have a situation where the nodes are seated on different domains in the network) we should check if any firewall is open for TCP protocols.
With this dead letter channel pattern we can configure all ESB worker nodes in the following way:
- ESB node 1 : Default to MB node 1, failover MB node 2, MB node 3.
- ESB node 2 : Default to MB node 2, failover MB node 3, MB node 1.
- ESB node 3 : Default to MB node 3, failover MB node 1, MB node 2.
The magic touch for making everything work together
The main reason why it was not possible to register a JMS proxy in a clusered ESB setup was because of a registration issue in ESB2MB. During deployment of the proxy, all individual ESB nodes will try to subscribe with the same subscription ID. This creates a JMS spec violation, because only one subscriber is allowed. JMS 1.1 does not support shared topic subscriptions by default. But since MB 3.0.0 this problem is solved. It will allow shared subscriptions. Note that JMS 2.0 will support shared topic subscriptions by default. So for next generation Message Brokers a working clustered setup can be done with less efford. For now configure in all MB nodes in <MB_HOME>/repository/conf/broker.xml
Also in every ESB node we should set the
Without setting the above parameter the message broker will wait forever for a signal from the message processor to process messages. And message delivery will not work. So setting these options message delivery is covered.
Please refer to further references for a more detailed description of these new features, and some working JMS proxy examples to try out.
The “after” scenario…
The Stock Exchange Market sends automated update messages through ESB services. From the services the ESB cluster sends the messages to the MB cluster. The MB cluster will then distribute the messages back to all the Stock Brokers with the help of ESB proxy services, or directly listening to Topics in MB.
All this is done in a stable, trusted and fast way.
Please don’t hesitate to contact us for more information or consultancy regarding this big subject. Thanks to Rob Blaauboer and Domenique Mulder for their contributions to this blog.
How to achieve delivery reliability with dead letter channel pattern
How to achieve delivery reliability with dead letter channel pattern part 2
WSO2ESB example two way request-response semantic JMS
Maximum Delivery Attempts to the Message Broker
Clustering MB 3.0.0
Clustering ESB 4.9.0