Our journey in enterprise integration highlights the evolution from the challenges faced with WSO2 Enterprise Integrator (EI) 6.5.0 to the streamlined efficiencies of WSO2 Micro Integrator (MI) 4.2.0. This transition exemplifies WSO2’s commitment to continuous improvement and innovation.
Navigating observability in System Stability
Welcome to a pivotal shift in enterprise integration, where observability becomes the beacon for system stability. Our narrative begins with the established JMX protocol, which has been a cornerstone in monitoring. However, with the advent of container orchestration technologies like Kubernetes, we’re ushering in a new era. This blog delves into the transformation from WSO2 Enterprise Integrator’s monolithic framework to a more dynamic, containerized approach with WSO2 Micro Integrator and JMX Exporter. Here, we’ll explore how modern tools like Prometheus and Grafana revolutionize our capacity to proactively manage and detect instabilities across WSO2 components, marking a significant departure from the bare-metal installations of the past.
The EI Experience: In-Depth Troubleshooting
Initially, in WSO2 EI 6.5.0, we faced significant startup delays, with times surging from 240 seconds to over 1,200 seconds. Our comprehensive troubleshooting process involved using tools like log4jdbc DriverSpy, JMX monitoring, and meticulous analysis of the wso2carbon.log file. The task of tailing and interpreting technical logs was a critical part of our diagnostic approach. This detailed analysis of log patterns was not only technically demanding but also time-consuming, often extending into days of work.
It underscored the robustness of WSO2’s tools in managing complex database connections and performance challenges, while also highlighting the intricate nature of identifying and resolving underlying issues in enterprise integrators.
This journey and the lessons learned were documented in detail in my previous blog post, “How to Fix a Slow-Starting WSO2 EI Server”, which further explores the depths of such technical endeavors.
Transition to MI 4.2.0: Enhanced Performance Tuning
With the upgrade to MI 4.2.0, we’ve fine-tuned the integration approach to enhance performance and monitoring capabilities:
1. Performance Tuning and Configuration:
- MI’s leaner architecture contributed to faster startups, showcasing its enhanced performance capabilities.
- We optimized the `deployment.toml` for effective memory and thread management, demonstrating MI’s flexibility:
2. Advanced Observability with JMX and Prometheus:
- Prometheus is utilized for real-time monitoring, showcasing MI’s advanced integration and observability capabilities.
- To enable detailed performance insights via JMX metrics, we introduce custom configurations in the
deployment.tomland set JVM command-line options for Prometheus compatibility:
```toml
[[synapse_handlers]]
name="CustomObservabilityHandler"
class="org.wso2.micro.integrator.observability.metric.handler.MetricHandler"
# Example datasource with JMX enabled
[[datasource]]
id = "StoreDB"
url = "jdbc:mysql://mysql:3306/store"
username = "root"
password = "root"
driver = "com.mysql.cj.jdbc.Driver"
jmx_enable = true
```
- JVM command-line options:
-DenablePrometheusApi=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999 - In the next step we will deploy a JMX Exporter as a sidecar to scrape JMX metrics and serve them in a Prometheus-friendly manner.
3. Docker Compose Setup for JMX Exporter:
- A JMX Exporter Docker container is set up to scrape metrics from the WSO2MI instance. Here’s the relevant snippet from the docker-compose file:
```yaml
jmx-exporter:
image: sscaling/jmx-prometheus-exporter
networks:
- app-tier
ports:
- "9072:9072"
environment:
SERVICE_PORT: 9072
JVM_OPTS: "-Djava.util.logging.config.file=/opt/jmx_exporter/logging.properties -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=5555"
volumes:
- ./conf/jmx-exporter/wso2mi_config.yml:/opt/jmx_exporter/config.yml
```- The
wso2mi_config.ymlfor the JMX Exporter is configured to connect to the WSO2MI JMX endpoint with the following settings:
```yaml
startDelaySeconds: 15
jmxUrl: service:jmx:rmi:///jndi/rmi://wso2mi:9999/jmxrmi
ssl: false
lowercaseOutputName: true
username: admin
password: admin
```- It’s crucial to use the correct
jmxUrlfor successful connectivity with WSO2MI’s JMX interface.
These steps illustrate the setup for enhanced monitoring capabilities in WSO2 Micro Integrator, enabling seamless integration with observability tools.
4. Visualizing MI Metrics with Grafana:
Configuring Grafana to display MI metrics enhances the observability of database performance and showcases the advanced capabilities of WSO2 Micro Integrator for in-depth analysis. With the use of the JMX Exporter, we have extended the default metrics to include a broader range of database connection parameters, allowing for a more comprehensive monitoring setup.
Instead of the previously mentioned db_active_connections and db_idle_connections metrics, we are now able to monitor an array of detailed metrics that give us insights into the database connection pool’s status and health. These metrics include:
[datasource id]_datasource_initialized: Indicates if the datasource has been initialized.[datasource id]_datasource_autocommit: Shows if auto-commit is enabled for the connections.[datasource id]_datasource_readonly: Reflects if the datasource is in read-only mode.[datasource id]_datasource_lastconnected: The timestamp of the last successful connection.[datasource id]_datasource_closed: Signifies whether the datasource has been closed.[datasource id]_datasource_lastvalidated: The timestamp when the connection was last validated.[datasource id]_datasource_maxageexpired: Indicates if connections have expired due to max age settings.[datasource id]_datasource_discarded: Counts the number of discarded connections.[datasource id]_datasource_suspect: Flags connections that might be suspect or potentially broken.[datasource id]_datasource_connectionversion: Version number of the connection.[datasource id]_datasource_timestamp: The current timestamp.[datasource id]_datasource_transactionisolation: Shows the transaction isolation level.[datasource id]_datasource_holdability: Indicates the holdability of the connections.[datasource id]_datasource_released: Counts how many connections have been released.
Creating Grafana dashboards that visualize these metrics allows administrators to monitor and analyze the performance and stability of their database connections in real time, providing actionable insights into the system’s operation.
Note: The metrics endpoint provided by WSO2 MI (http://<MI_HOST>:<MI_PORT>/metrics) does not expose these JMX metrics by default. Using the JMX Exporter as outlined in my Docker Compose setup, we’re able to scrape these detailed JMX metrics from WSO2 MI and serve them in a format that Prometheus can consume and Grafana can visualize.
Uncovering the Root Cause with MI’s Advanced Monitoring
MI’s advanced monitoring features significantly streamline the process of identifying examples like the MS SQL database server as the primary issue talked about in my previous blog. The integration of Prometheus and Grafana offers robust solutions for real-time analysis:
- Rapid detection of anomalies in database performance metrics became straightforward, allowing for immediate identification of unusual patterns or extended usage.
- Continuous monitoring shed light on external factors impacting database functionality, revealing potential underlying issues.
- Direct and comprehensive insights into database operations facilitated a more efficient approach to diagnosing and addressing problems, potentially pinpointing challenges associated with the Windows server hosting the database.
These enhanced monitoring capabilities in MI underscored the efficiency and effectiveness of the platform in swiftly identifying and addressing external database-related challenges.
Reflecting on EI and MI Approaches
The transition from EI’s manual monitoring to MI’s automated performance management reflects a significant advancement in integration technology. This shift emphasizes WSO2’s dedication to providing solutions that reduce complexity and enhance efficiency.
Conclusion: Embracing the New Era with MI 4.2.0
Our transition from WSO2 Enterprise Integrator to Micro Integrator isn’t merely a change in technology — it’s a leap forward in our integration strategy. WSO2 Micro Integrator 4.2.0, equipped with its automated monitoring tools and rich diagnostics, heralds a new chapter in integration solutions. Here, we’ve addressed crucial technical challenges, paving the way for:
- Effective observability of WSO2 Micro Integrator using Prometheus and custom Grafana dashboards.
- Elimination of the need for heavy-weight processes for JMX monitoring, optimizing overall resource consumption.
- A secure, flexible, and efficient setup using containers, opening doors to Kubernetes for orchestrating our integration landscape.
These enhancements are integral to sustaining robust, scalable, and flexible platforms, crucial for pinpointing and resolving the complexities of integration challenges.
