Data Integration: How to Connect Business Data Across Systems

Data integration is the work of bringing data from separate business systems into one consistent view that people can use. AWS’s explanation describes it as the process of achieving consistent access and delivery for all types of data in the enterprise. In AWS’s wording, it includes architectural techniques, tools and practices that unify disparate data for analytics. This guide gives the order of work in seven steps, from business questions to a named owner for each feed, with a short section on master data management. It sits under the Signisys use case on connecting business data across systems.

Data Integration in One Plain Line

Picture sales in one application, stock in another and payroll in a third. A manager then asks one question that needs all three at once. Seen that way, data integration can be treated as a set of decisions before it is software. The decisions cover which questions to answer, which systems to draw on, whose definitions to use and how data moves. Tools come in only after those are settled.

Three terms recur below:

• Source system: an application the data comes from, such as the order or payroll application.
• Pipeline: the route data takes from a source system to the shared place.
• Master data: in this article, the shared core records for customers, products and suppliers.

This guide does not explain why two reports on the same measure show different figures. For background on why separate systems hold separate data, see the earlier post.

Data Integration Strategy: Steps 1 to 3, Decide Before You Connect

In Signisys’ working description, the work runs in seven steps, and this order is our own method rather than a standard. The first three steps are decisions made on paper, before anyone chooses software. A data integration strategy, in this guide, is the written result of those three steps. Steps 4 to 7 then put it into practice.

Data Integration Step 1: Name the Business Questions

Write down a few questions that the combined data has to answer. Each question then names the data it needs. For example, “Which products sold out last month, and which customers were waiting for them?” needs orders, stock and customer records.

Keep the list short at the start, because each new question can add sources, owners and definitions to agree. A question nobody asks yet can wait for a later round.

Ask yourself this: if a question will not fit in one sentence, what would the finished report show? If the answer is unclear, the question needs more work before any data moves.

Data Integration Step 2: List Source Systems and Who Owns Each

For each system a question needs, write four things down. Note what it holds, who owns it, how data can leave it and whether the owner agrees to share it.

Add one row per application. Where an owner says no, the question waits until someone senior settles it. In the meantime, record the refusal next to the system. Nobody should connect to it unannounced.

Data Integration Step 3: Agree One Definition per Shared Term

Terms that several systems share need one meaning in the combined data. Customer, order and revenue are the obvious candidates. For example, is a cancelled order still an order? Does revenue include returns?

Settle each answer with the owners named in step 2. Then keep a one-line definition for each term on a page the whole team can open. From then on, the combined data follows those definitions, not the habits of any one application.

Customers, products and suppliers need one record each as well. That part is master data management, covered in its own section below.

Data Integration Steps 4 and 5: Move and Load the Data

With the decisions written down, the next two steps carry the data from each source to one shared place. Each flow gets its own choice of method.

Data Integration Step 4: Choose How Data Moves for Each Flow

A flow is one route from one source system to the shared place. Three ways to move data come up here:

• Scheduled batch: a copy taken at set times, such as each night.
• Near real time: changes sent soon after they happen.
• Direct connection: the report reads the source when someone asks.

AWS describes two methods close to the third way. In its wording, data virtualization does not move data between systems but creates a virtual unified view, and data federation creates a virtual database on top of several sources and fetches the data when a query arrives. AWS also describes replication, which creates duplicate copies of data instead of moving data physically from one system to another.

In Signisys’ working description, the choice per flow rests on one question. How old may the figure be when someone uses it? If a day is fine, a nightly batch is enough. But a figure needed within the hour points to near real time or a direct connection.

Whether to reshape data before or after loading it is the ETL versus ELT choice, which this guide does not cover.

Data Integration Step 5: Load to One Shared Place

Send each flow to one store that all the reports read from. That store can be a data warehouse, a lake or something similar. Here, a warehouse means a database organised for reporting, while a lake keeps files in their original form.

Only the pipelines write to that store. Nobody edits it by hand, because a later load may overwrite the edit.

Master data management meets this step at the load. Each customer, product and supplier should arrive with its agreed ID. Records from different sources then line up.

As one platform example, Microsoft Learn’s Azure Data Factory introduction describes it as a managed cloud service built for complex hybrid ETL, ELT and data integration projects. The same page points people new to data integration to Data Factory in Microsoft Fabric. This guide names Azure Data Factory as an example only, not as a recommendation.

Data Integration Steps 6 and 7: Check It and Keep It Running

The last two steps test the combined figures and keep the connections running after launch. Both start once the first load has run.

Data Integration Step 6: Check Combined Figures Against Source Reports

Before anyone relies on the combined data, pick one closed period, such as last month. Then compare its totals with the reports each source system already produces.

In Signisys’ working description, each gap gets a written explanation before the data goes into use. A gap may come from a definition agreed in step 3, a flow timed in step 4 or a fault in a pipeline. Each of those has a different fix. For that reason, keep the three apart. Repeat the comparison after each change to a source or a pipeline.

For cleaning values and setting data rules, see the use case on improving data quality.

Data Integration Step 7: Assign an Owner and a Failure Alert

Name one person who owns each connection, with a deputy. Set an alert that tells that owner the day a feed stops. That way, nobody has to wait for an odd-looking report to find out.

Keep a short log as well: the date, what changed in the source or the pipeline, and who made the change. When a source system changes, its owner should tell the connection owner first.

This guide stops where the shared data is ready. Building reports on top of it is a separate job. For that side, see the management reporting use case.

Master Data Management: One Record for Each Customer, Product and Supplier

In this article, master data means the shared core records for customers, products and suppliers. Master data management, in turn, means keeping one agreed record for each of them. It is part of steps 3 and 5 rather than a separate project. Three habits keep it workable:

• Give each record an agreed ID, to match the same customer across sources.
• Give each record type one owner, for example sales for customers and purchasing for suppliers.
• Merge duplicates by a written rule, not by hand.

Start with the record type that your first business question depends on. Then add the others as later questions need them.

What This Article Leaves Out, and When to Get Help

This guide covers the order of work. Master data management gets only the part that steps 3 and 5 need. It leaves out why two reports disagree, a scored checklist and the ETL versus ELT choice. Cleaning and data rules sit with the data quality use case, and report building sits with the management reporting one. Connecting applications and processes, beyond moving data, is a different job. For that, see connecting applications and processes.

Bring in specialist help when one of these is true:

• The owner of a source system is out of reach.
• Several source systems feed one question.
• No one is able to write the definition of a shared term.

Signisys provides services for connecting and analysing business data through its business intelligence and analytics solution and its data and AI services. Read this paragraph as a description of a Signisys service, not neutral advice.

Frequently Asked Questions