Non-aligned Information Systems
Every company uses several information systems to host its business operations. In the ideal world, these systems are aligned by Data Integration. In most cases, these information systems, like a CMS, ERP, and a webshop, are not optimally aligned with one another. For instance, systems are not completely standardized, or not all standardized at the same level, or information connections aren’t even in place between systems.
In order to align these information systems, the choice to use a Canonical Data Model (CDM) is commonplace. This results in an immediate question regarding to what extent and to what level of detail a CDM needs to be defined. An overly detailed CDM isn’t manageable, but a CDM that is too abstract and superficial isn’t practical either. The real challenge lies in creating a balanced, well thought-through CDM, in which exactly the right amount of detail is included.
As soon as the CDM is defined, regardless of the level of abstractness and depth of the defined details, the immediate next question will be: how would you like your information systems to exchange data with one another? There are dozens of hurdles to be overcome, each of them comprising a challenge for your IT-department.
Import and export formats are an example of such a challenge. Major differences can be observed between cases concerning modern information systems supporting RestServices, compared with somewhat old-fashioned systems which purely rely on SOAP formats or systems that only recognize one file interface.
Beyond the differences in format exchange, differences in versions of information systems could form the next hurdle. One example is found in the consequences of updates and upgrades for information systems, as well as the ongoing improvements within organizations leading to replacement of certain systems. After an update, upgrade, or replacement of a system, the new set-up needs to remain compatible with the way data management is done within the IT landscape; which presents the next challenge.
Real Time or Batch Oriented
Another challenge IT-Managers face is the gap they need to bridge between real time and batch oriented data processing. Say an organization requests real time information, regardless of the fact that the underlying information system only supports batch oriented data processing, the gap will need to be bridged from a technical perspective.
To make sure that a real time information system is able to exchange data with a batch oriented information system, a mechanism to guarantee the bidirectional data exchange is needed. Of course, there are technical solutions on the market, like message queuing of event streams via Kafka technology, but they inevitably lead to the next challenge of finding a suitable integration pattern to implement these solutions.
Data Integration with a Single or Multiple Source of Truth
In the past, we learned to work towards a single source of truth by using data warehouses. It turned out to be more complex in practice; making this classical approach of data warehouses not the optimal solution for every situation. We will have to deal with the fact that there may exist more parallel truths in different information systems within an organization. Therefore, it is a challenge to make each of these parallel sources of truth available at the right time to an information system, or person, within the organization looking for the respective information.
The most effective solution to this challenge is an integration pattern based on APIs, underlying Micro Services and, for example, Kafka Streams. Within this pattern, several information sources will need to be queried in real time, and received results will have to be consolidated in a so-called compositive service. By use of this architectural principle, each underlying information system exists in its own right, and will not undermine the integrity of any higher source of truth.