One of the prominent Enterprise Integration Patterns in API Integration is the Scatter and Gather Pattern. It involves collecting multiple responses from backends and consolidating them into a single output accessible, oftentimes via the frontend.
In this article, we’ll explore a scenario where we gather quotes for the hourly rate for colour-washing houses from APIs of various providers, consolidate them, and present in a unified view at the frontend. We are going to do this with Apache Camel on top of Spring Boot.
For a better understanding of the Scatter and Gather Pattern, you can refer the document here.
Initial Project Setup
To generate the initial project, we can use the Spring Boot Initializer. Let’s use the below selection for project generation. Under dependencies you can see that I have selected Apache Camel.
Once we select the Apache Camel dependency, it will include the below dependency in pom.xml
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-spring-boot-starter</artifactId>
<version>4.4.1</version>
</dependency>
For the build in this article will be using Apache Maven 3.8.6 and Java version: 18.0.2 in Apple M1 Pro, targeting Java Version 11.
To gain more insights on the project structure you can refer previous article on Apache Camel Part-1 and Part-2 as well.
High Level Integration Flow Diagram
As illustrated in the diagram above, we’ll have a front-end API called GET /quotes/painting. This API will concurrently invoke the three vendor APIs, retrieve the response from each, aggregate, and then send back a consolidated response to the client.
Implementation
Understanding the pom.xml
| No. | Dependency | Description |
| 1 | camel-spring-boot-starter | Helps integration of Camel routes within the Spring Boot Applications |
| 2 | camel-rest-starter | To support Camel Routes to configure as Restful Services. |
| 3 | lombok | This is for ease of development like slf4j logging. Provides getters setters by simply using annotations. |
| 4 | camel-netty-http | Camel supports rest components such as jetty, servlet, undertow and here I’m using netty. |
| 5 | Properties -> java.version | Defining 11, so the compatibility with JDK 11 will be confirmed and the class files will be generated for the specific major version. |
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.2.3</version>
<relativePath/>
</parent>
<groupId>com.quotes</groupId>
<artifactId>PaintingServiceQuotes</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>PaintingServiceQuotes</name>
<description>Project for Painting Service Costs</description>
<properties>
<java.version>11</java.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-spring-boot-bom</artifactId>
<version>4.4.1</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>3.2.3</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-rest-starter</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-netty-http</artifactId>
<version>4.4.1</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
Spring Boot Main Application Class Implementation
package com.quotes.painting;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class PaintingServiceQuotesApplication {
public static void main(String[] args) {
SpringApplication.run(PaintingServiceQuotesApplication.class, args);
}
}Backends API Route Builder
We are implementing the Backends API Route builder to have the vendor specific APIs. You can see, we have 3 – Vendor APIs defined, and they will be sending back an application/json responses.
package com.quotes.painting.builders.api.back;
import lombok.extern.slf4j.Slf4j;
import org.apache.camel.builder.RouteBuilder;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
@Slf4j
@Component
public class BackApiBuilder extends RouteBuilder {
@Value("${custom.netty.http.port}")
private int nettyHttpPort;
@Override
public void configure() throws Exception {
log.info("BackApiBuilder Route Builder Initiated.");
restConfiguration()
.component("netty-http")
.port(nettyHttpPort);
// 1. API of the Vendor One
rest("/quotes")
.get("/vendor-one")
.produces("application/json")
.to("direct:vendorOne");
from("direct:vendorOne")
.log("Received GET request for vendorOne.")
.setBody(constant("{\"vendor\": \"vendorOne\", \"serviceType\": \"Painting\", \"Hourly\": \"25$\"}"));
// 2. API of the Vendor Two
rest("/quotes")
.get("/vendor-two")
.produces("application/json")
.to("direct:vendorTwo");
from("direct:vendorTwo")
.log("Received GET request for vendorTwo.")
.setBody(constant("{\"vendor\": \"vendorTwo\", \"serviceType\": \"Painting\", \"Hourly\": \"35$\"}"));
// 3. API of the Vendor Three
rest("/quotes")
.get("/vendor-three")
.produces("application/json")
.to("direct:vendorThree");
from("direct:vendorThree")
.log("Received GET request for vendorThree.")
.setBody(constant("{\"vendor\": \"vendorThree\", \"serviceType\": \"Painting\", \"Hourly\": \"15$\"}"));
}
}
Front API Route Builder
Following class holds the main implementation of the scatter and gather pattern logic. In this sample I have used the multicast camel component. By default, multicast works on a single thread, however parallel processing can also be enabled. Also “AggregationStrategy” has been used to combine the response payloads and send back as consolidated JSON Response.
package com.quotes.painting.builders.api.front;
import com.quotes.painting.aggregaters.JsonAggregator;
import lombok.extern.slf4j.Slf4j;
import org.apache.camel.builder.RouteBuilder;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
@Slf4j
@Component
public class FrontApiBuilder extends RouteBuilder {
@Value("${custom.netty.http.port}")
private int nettyHttpPort;
@Override
public void configure() throws Exception {
log.info("FrontApiBuilder Route Builder Initiated.");
restConfiguration()
.component("netty-http")
.port(nettyHttpPort);
rest("/quotes")
.get("/painting")
.produces("application/json")
.to("direct:paintingQuotes");
from("direct:paintingQuotes")
.log("PaintingQuotes Request Initiated.")
.multicast()
.parallelProcessing()
.aggregationStrategy(new JsonAggregator())
.to("netty-http:http://localhost:8090/quotes/vendor-one")
.to("netty-http:http://localhost:8090/quotes/vendor-two")
.to("netty-http:http://localhost:8090/quotes/vendor-three")
.end()
.log("Process Completed: ${body}");
}
}
Aggregation Strategy Implementation
This class helps to aggregate the responses of the back-end APIs.
package com.quotes.painting.aggregaters;
import org.apache.camel.AggregationStrategy;
import org.apache.camel.Exchange;
import org.springframework.stereotype.Component;
@Component
public class JsonAggregator implements AggregationStrategy {
@Override
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if (oldExchange == null) {
return newExchange;
}
String oldBody = oldExchange.getIn().getBody(String.class);
String newBody = newExchange.getIn().getBody(String.class);
String body = null;
if (!oldBody.startsWith("[")) {
body = "[ " + oldBody + ", " + newBody + " ]";
} else{
body = oldBody.replace("]", "") + ", " + newBody + " ]";
}
oldExchange.getIn().setBody(body);
return oldExchange;
}
}
Resources/application.yml Configuration
camel:
springboot:
name: PaintingServiceQuotes
logging:
level:
org:
apache:
camel: INFO
custom:
netty:
http:
port: 8090
Build and Test
Execute the typical maven build command at the project home directory
mvn clean installThen execute the below command to start the Spring Boot Application.
java -jar target/PaintingServiceQuotes-0.0.1-SNAPSHOT.jarUsing a client or curl command invoke the front-end API, which will respond with the aggregated response.
That’s concludes the discussion on how we can implement the scatter and gather Enterprise Integration Pattern with Apache Camel. Hope this step-by-step guide provided a detailed explanation on how we can achieve this with Camel.
Hope to catch you in another blog. Stay tuned for more insights and updates.
