How to Improve Data Quality for Analytics and AI

A sales head who quietly rechecks a report before a review meeting has already spotted the problem. What is missing is an order in which to tackle it. In this guide, how to improve data quality comes down to five steps that a mid-size firm can begin with one report and two people. Pick the figures that matter and trace where their errors come in. Then write a rule for each key field, change the place where someone creates each value, test a sample and give each field an owner. Each step also ends with a short note to keep. The guide belongs to the Signisys use case on reliable data for analytics and AI.

How to Improve Data Quality: Two Things to Have First

Start with one report or AI use the firm wants to rely on, and two people: one who reads the figures and one who knows where the team keeps the records. The first two steps need nothing more.

A few terms, defined once:

• Data quality, in this guide, is how well a firm’s records fit one named use, such as a monthly report or the answers an AI assistant gives. This follows the general reading of data quality as fitness for intended use, as described in Wikipedia’s entry on data quality.
• A key field is a single column that a report, or an answer from AI, leans on, for example a depot, a customer name or a return date.
• Analytics is the work of answering business questions from records, in reports and dashboards.
• Artificial intelligence, here, means software trained on examples to answer, sort or predict.

Why two people and not one? The reader of the report knows which number looks wrong. The keeper of the records, on the other hand, knows where that number begins. As a result, each may see a half the other misses.

Agree, as well, what “good enough” means for the chosen report before anyone fixes anything. A monthly report and an AI assistant’s answers may need different standards, and the firm sets each one. This is Signisys’ reasoning and not a cited rule.

If your firm wants a name for the written order of work and the owners it ends with, data quality framework fits. The label matters less than having the report and the two people ready.

How to Improve Data Quality: The Sequence to Follow

Work through five steps in this order. The order comes from Signisys’ own working method and is no published standard. The sections below take one step each and end with the note to keep when you finish it.

Fixing sits fourth on purpose. Until the team knows which fields matter, where they start and what a good value looks like, it has nothing to aim a clean-up at.

The sequence is reasoning for a first pass, and it makes no promise about the outcome. A team may loop back to step 2 when step 5 turns up a field that nobody traced.

Choosing the Figures That Matter and Finding Where Errors Enter

Begin with people rather than systems. Sit down with whoever reads the report and ask one question: which figures on this page would you hesitate to defend in front of the board? Write those down. Then list the fields that feed each one. For instance, a weekly count of stock on hand by warehouse rests on an item code, a warehouse and a quantity.

Note after step 1: the figures chosen, the fields behind them and who picked them.

Step 2 takes each field back to where its values begin. Ask three questions about each field you listed:

• Who types this value?
• Which file or import brings it in?
• What does the screen allow a person to enter?

Write one line per field, saying where its values start and who can look at that place. When two teams fill the same field in different ways, give it a line for each team. A branch in Pune might key dates day-first while a branch in Dubai imports them from a spreadsheet laid out month-first, for example.

Note after step 2: each field with its starting point and a named contact.

Nobody corrects anything until step 4. The first two steps are conversations, and what they produce is a map that the later steps work from. Keeping them short also matters. The aim is one report and its fields, and not the other tables the firm holds. The ways a record can go wrong are laid out in a short catalogue of five kinds of fault and who to ask about each.

Writing a Rule for Each Key Field

A rule, in Signisys’ working method, is a single sentence that a colleague can check by looking at one value. If two people could read it two ways, it needs another draft.

Rules can cover the format of a value, its unit, the values allowed and whether the field may stay empty. Here are three examples, each with the person who tests it:

• Customer name: spelt as on the customer’s registration certificate, and entered once. The sales coordinator tests it.
• Depot: chosen from the firm’s list of depots, not typed in and not left empty. Tested by the report owner.
• Item weight: recorded in kilograms, with no other unit. The stores lead checks it.

Write each rule with both people from step 1 present. The reader of the report knows what a usable value looks like, while the keeper of the records knows what the form can hold to. The examples carry no threshold or percentage. Instead, the firm sets those for itself.

These three show how to word a rule. They are not a standard to copy, and your own key fields will need their own sentences.

Note after step 3: each key field, its rule and whoever can test it.

Fixing at the Source, Then Correcting What Is Stored

Change the place where someone creates a value first, and only then correct the records already stored. The order matters: an unchanged form can let the next person type the same error back in.

At the source, the change can be small. Make a field required, swap a free-text box for a list of allowed values, or ask the team that sends a monthly file to use one date style.

With the source changed, turn to the stored records. Start with the fields named in step 1 rather than the whole database. After all, those are the fields the report reads. Remove duplicate entries, and fill gaps where a trusted record exists elsewhere, such as the signed agreement. Where the errors come from two customer lists merged into one, the work overlaps with bringing business records together.

Two questions decide who acts. Who may change this form? And who signs off a correction to stored records? Put a name against each before anything changes.

Note after step 4: what changed at the source, what the team corrected in stored records, and who signed it off.

Checking the Result and Giving Each Field an Owner

Test a sample of records against the rules, write down what you found, and give each key field a named person. That closes the first pass.

Draw the sample from records created after the source change, as well as from corrected ones, to cover both halves of step 4. For each value the question is plain: does it meet its rule or not? Then record the answer, who looked and the date.

Then name an owner for each key field. Say what that person does when a value breaks its rule: whom they tell, where they log it and who can change the form. The sample size, and how frequently to repeat the check, are for the firm to set; this guide gives neither. How later rounds might be run is compared in the places a data rule can be checked and who runs each.

Note after step 5: the sample, what turned up, who checked, and who owns each field.

Together with the four earlier notes, this one lets a successor see what the team decided, by whom and why. To score a sample with marks and dated evidence, use a ten-check list for running your own test of one data set.

How to Improve Data Quality: Following a Hire-Returns Report

We invented the equipment-hire firm below to show the five steps end to end. No client or Signisys engagement lies behind it.

Its operations head has stopped trusting the weekly report of hire returns by depot. With the depot-systems lead, she picks that report for step 1 and lists its fields: the hire date, the return date and the depot. Tracing them in step 2, the pair find that counter staff type the return date by hand at each depot, while the hire date comes straight from the booking screen.

For step 3 they write one rule: staff enter a return date as day, month, year, and it falls on or after the hire date. Step 4 starts at the counter, where the form gets a date picker. Only after that does the team correct stored returns, beginning with the largest depots.

In step 5 they test a sample of new and corrected returns against the rule, then name the depot-counter lead as owner of the return date.

For this firm, the answer to how to improve data quality ended up as five short notes and one named person for the field that mattered. Your own report and fields will differ, and the example offers no verdict on them. When the fields can be relied on, the next job is described in five moves that take the copying out of a recurring report.

When the Sequence Needs Outside Eyes

Three choices remain with the firm after the five steps. Which fields deserve attention first? How far back should the team correct stored records? And how much change can its forms, imports and teams absorb at once? Each of those turns on the firm’s own records and staff.

An outside review earns its place at one point. That is when the trail from step 2 leads to a field filled by two or more systems, and the people who run those systems disagree about which value is right.

Signisys works with firms on reports, analytics and AI; see the reporting and analytics solution and the data, analytics and AI services. The consultation below is a Signisys service, offered as such and not as neutral advice. Bring the report you doubt and the notes from steps 1 and 2.

FAQ on How to Improve Data Quality