In the realm of distributed systems and enterprise integrations, monitoring application logs becomes critical for maintaining the stability, performance, and security of services. WSO2 Micro Integrator is a powerful integration runtime built for microservices architecture. It plays a crucial role in executing business-critical integrations, handling complex workflows, and enabling API interactions. Like any service-based environment, troubleshooting issues within the Micro Integrator requires a robust log monitoring solution.
Log monitoring offers teams deep insights into system behavior, helps detect issues early, and addresses performance bottlenecks before they impact operations. For WSO2 Micro Integrator, collecting, monitoring, and analyzing logs in a seamless and organized manner ensures smoother operations and faster troubleshooting, enhancing overall service reliability.
By integrating WSO2 logs with Grafana, users can visualize these logs, set alerts, and even correlate logs with other metrics for a comprehensive view. In this post, we will focus on how to leverage Grafana Alloy to scrape WSO2 Micro Integrator logs, send them to Grafana Loki, and ultimately display them in Grafana’s UI for efficient monitoring and troubleshooting.
What are Grafana, Loki, and Alloy?
Before diving into the configuration, let’s break down the key components that will power our logging solution:
– Grafana: Grafana is a widely-used open-source platform for monitoring and observability. It allows users to visualize metrics, logs, and alerts through customizable dashboards. With its ability to integrate with various data sources (e.g. Prometheus , Loki , MySQL etc.), Grafana is a go-to tool for monitoring applications in real time.
– Loki: Loki is a horizontally scalable, highly available, and multi-tenant log aggregation designed by Grafana Labs. Unlike traditional log management systems, Loki is optimized for cost-efficiency by only indexing metadata, not the logs themselves.
– Grafana Alloy: Alloy is a log collection agent designed to scrape logs from different sources (such as Docker containers, Kubernetes pods, or files) and ship them to Grafana Loki. It simplifies the task of log scraping by providing an efficient and configurable way to send logs to Loki, making it ideal for monitoring containerized environments like Podman.
Configuring Grafana Alloy to Scrape WSO2 Micro Integrator Logs
Now that we understand the tools involved, let’s dive into how to configure Grafana Alloy to collect logs from WSO2 Micro Integrator running in a Podman container environment and publish them to an existing Loki and Grafana setup.
Step 1: Configure and Install Grafana Alloy
Configuration File: Create a configuration file (config.alloy) to instruct Alloy on how to collect and forward logs.
This Alloy configuration defines how logs from the WSO2 Micro Integrator running in a Podman environment will be scraped and sent to Loki. Let’s break down the configuration step by step:
Log File Matching: This block defines the log files to be scraped.
- path_targets: This is an array of log file paths and associated metadata labels that describe the environment.
- __path__: The actual path to the log file on the system (e.g., /tmp/mi-logs/wso2carbon.log).
- Metadata fields like Environment, ServiceName, and Node provide context to the logs.
- “Environment” = “Dev”: Specifies that these logs are from a development environment.
- “ServiceName” = “Micro Integrator”: Indicates that the logs are from the WSO2 Micro Integrator service.
- “Node” = “MI-Node1”: These logs are coming from a specific node in the
yourenvironment (e.g., “MI-Node1”).
Log Scraping Configuration: This block instructs Alloy to scrape the log files specified in the local.file_match.milogs block.
- targets: The path_targets from the earlier local.file_match “milogs” block are referenced here. Alloy will scrape the logs from these files.
- forward_to: This tells Alloy where to send the logs after scraping. In this case, the logs are forwarded to the Loki write endpoint defined in the next section (loki.write “local_loki”).
Sending Logs to Loki: This block configures Alloy to send the scraped logs to a specific Loki instance.
- url: This is the URL for the Loki server’s log ingestion API. The logs will be pushed to the Loki server at http://<grafana-loki-host>:3100/loki/api/v1/push.
- Replace <grafana-loki-host> with the actual hostname or IP address of your Grafana Loki instance.
- Mount Log Files: Make sure the path to your WSO2 Micro Integrator log files is correctly mounted in the Podman container running Alloy. In this case, /opt/wso2/apps/logs/mi/ will be the directory where your WSO2 logs are stored in host machine.
- Mount Configuration: Make sure you mount the configuration created in Step 1 to container path /etc/alloy/config.alloy. In this case, mount the host path /opt/wso2/apps/alloy/config.alloy to container path /etc/alloy/config.alloy
- Install Alloy: Download and install Grafana Alloy container for your environment. The installation will depend on whether you’re using Podman or Docker containers. For a Podman environment, you can run Alloy in a container as follows.
Step 2: Configure Loki to Receive Logs
your Loki instance is active and configured to receive logs from Grafana Alloy. Please note that this post focuses on configuring Alloy for log scraping and does not cover the setup of Loki and Grafana themselves.
Step 3: Viewing Logs in Grafana
Add Loki as a Data Source: In Grafana, add Loki as a data source by navigating to Configuration > Data Sources > Loki and provide the URL where Loki is running (e.g., http://localhost:3100).
Create Dashboards: Once Loki is configured, you can create custom dashboards in Grafana to view the WSO2 logs.In Grafana, filter logs, set alerts, or add annotations to detect patterns or anomalies effectively.
Explore Logs: Use the “Explore” feature in Grafana to query and troubleshoot specific log events. This makes it easy to pinpoint errors or monitor log streams in real-time.
Summary
Monitoring WSO2 Micro Integrator logs through Grafana offers a powerful and visual way to track the health and performance of your integration services. By using Grafana Alloy to scrape logs, Loki to store them efficiently, and Grafana to explore and visualize them, you can gain comprehensive insights into your application’s behavior and troubleshoot issues quickly.
This blog covers configuring Grafana Alloy in a Podman container environment and visualizing logs in Grafana. Such a setup ensures seamless log aggregation and monitoring, allowing you to stay ahead of issues and ensure optimal service performance.
Do you have questions about Podman, Grafana, Loki, Alloy, or WSO2 Micro Integrator? Our experts at Yenlo are ready to help you get the most out of your integration setup.
