Your challenge

Data Integration

Data Integration is one of the biggest challenges for many organizations nowadays. It all comes down to the complexity of data exchange and the multitude of factors that need to be considered for an effective and efficient setup.

IMG 4583 600x350 1

Non-aligned Information Systems

Every company uses several information systems to host its business operations. In the ideal world, these systems are aligned by Data Integration. In most cases, these information systems, like a CMS, ERP, and a webshop, are not optimally aligned with one another. For instance, systems are not completely standardized, or not all standardized at the same level, or information connections aren’t even in place between systems.

In order to align these information systems, the choice to use a Canonical Data Model (CDM) is commonplace. This results in an immediate question regarding to what extent and to what level of detail a CDM needs to be defined. An overly detailed CDM isn’t manageable, but a CDM that is too abstract and superficial isn’t practical either. The real challenge lies in creating a balanced, well thought-through CDM, in which exactly the right amount of detail is included. 

Technical Incompatibility

As soon as the CDM is defined, regardless of the level of abstractness and depth of the defined details, the immediate next question will be: how would you like your information systems to exchange data with one another? There are dozens of hurdles to be overcome, each of them comprising a challenge for your IT-department.

Import and export formats are an example of such a challenge. Major differences can be observed between cases concerning modern information systems supporting RestServices, compared with somewhat old-fashioned systems which purely rely on SOAP formats or systems that only recognize one file interface. 

Beyond the differences in format exchange, differences in versions of information systems could form the next hurdle. One example is found in the consequences of updates and upgrades for information systems, as well as the ongoing improvements within organizations leading to replacement of certain systems. After an update, upgrade, or replacement of a system, the new set-up needs to remain compatible with the way data management is done within the IT landscape; which presents the next challenge.

Real Time or Batch Oriented

Another challenge IT-Managers face is the gap they need to bridge between real time and batch oriented data processing. Say an organization requests real time information, regardless of the fact that the underlying information system only supports batch oriented data processing, the gap will need to be bridged from a technical perspective.

To make sure that a real time information system is able to exchange data with a batch oriented information system, a mechanism to guarantee the bidirectional data exchange is needed. Of course, there are technical solutions on the market, like message queuing of event streams via Kafka technology, but they inevitably lead to the next challenge of finding a suitable integration pattern to implement these solutions.


Data Integration with a Single or Multiple Source of Truth 

In the past, we learned to work towards a single source of truth by using data warehouses. It turned out to be more complex in practice; making this classical approach of data warehouses not the optimal solution for every situation. We will have to deal with the fact that there may exist more parallel truths in different information systems within an organization. Therefore, it is a challenge to make each of these parallel sources of truth available at the right time to an information system, or person, within the organization looking for the respective information. 

The most effective solution to this challenge is an integration pattern based on APIs, underlying Micro Services and, for example, Kafka Streams. Within this pattern, several information sources will need to be queried in real time, and received results will have to be consolidated in a so-called compositive service. By use of this architectural principle, each underlying information system exists in its own right, and will not undermine the integrity of any higher source of truth. 

More on Data Integration?

Are you interested in learning more about Data Integration and the solutions we offer? Schedule a consultancy session with one of our integration experts.

Plan a meeting

How can we help you?

Healthcare icon

WSO2 Consultancy & Health Checks

Capacity and WSO2 expertise, specifically adapted to the needs of each individual client.
Learn more
WSO2 Installation Configuration

WSO2 Enterprise and Solution Architecture

Together, our experts design an architecture that answers to all your business and IT needs.
Learn more
WSO2 Migration Services

WSO2 Migration Services

Let Yenlo migrate you to WSO2 and we guarantee you a cost-saving of 20-40% per year!
Learn more
WSO2 API Manager 1

WSO2 Installation & Configuration

Yenlo’s WSO2 Installation & Configuration service helps you find the perfect alignment you were looking for.
Learn more
WSO2 Software Development

WSO2 Software Development

In this phase, we help you with the installation and configuration of your middleware layer.
Learn more
WSO2 API Security

WSO2 API Security via 42Crunch

Audit, Scan and Protect your APIs from code to production by empowering developers with the right tools.
Learn more
WSO2 Managed Services Support

WSO2 Managed Services & Support

We offer support in pretty much every way possible, for as long as you need it.
Learn more
WSO2 Training Certification

WSO2 Training & Certification

If you want to start using WSO2 or use your WSO2 products to their full potential.
Learn more

Get in touch, contact us!

Get in touch and contact us by submitting the contact form below.
Article notification
Privacy Policy*
This field is for validation purposes and should be left unchanged.