A good data platform: why it pays off and why integration is more than ETL

Every organisation has its data spread across systems: an ERP, a CRM, a webshop, an HR system, a planning tool and a fair number of Excel files. That in itself is not a problem. The problem starts the moment someone asks a question that spans more than one system. How much revenue do we make per customer, including returns and service costs? It then turns out that the customer has a different name in the CRM than in the ERP, that returns live in a separate system and that service costs are tracked in a spreadsheet. Three people spend a week on it and the answer is almost right.

A data platform is where that problem gets solved structurally. This article explains what it delivers to your organisation, why ‘integration’ is more than the overnight data processing most people know it as and which agreements you need before the technology starts to help.

What a data platform is, without the technology

Think of a distribution centre. Goods arrive from dozens of suppliers, each with their own boxes, labels and pallets. At the entrance they are checked, given a standard label and placed in a fixed location. From that moment on, everyone in the building knows what is there, how much, and from whom. The shops order from the distribution centre, not from the individual suppliers.

A data platform does the same with data. The source systems are the suppliers. The platform checks what comes in, gives it one meaning (a ‘customer’ is the same customer everywhere) and makes it ready for whoever needs it: reports, other applications and, increasingly, AI applications. The databases, pipelines and cloud services needed for that are the building and the shelving. Useful, but the platform only becomes valuable through the agreements on labels, checks and ownership.

What it delivers to the business

Fewer arguments about numbers. When revenue, margin and customer counts are calculated in one place and by one definition, the familiar meeting in which three departments each bring a different figure disappears. Decisions move faster because the conversation is about the substance and not about who has the right file.

New questions answered sooner. As long as every question means someone has to stitch together data from four systems, a new analysis takes weeks. Once the data sits standardised on the platform, a new report or a new model is a matter of days. The gain is not in the first question but in the tenth.

Provenance you can show. The GDPR, the EU AI Act and your own auditor want to know where a figure comes from and who is responsible for it. On a well-designed platform that can be traced (data lineage) rather than reconstructed from emails.

A foundation for AI. An AI model trained on contradictory or incomplete data gives confident wrong answers. A platform with checked, unambiguous data is the precondition for taking AI beyond a pilot. We wrote earlier about what the EU AI Act asks of you.

Less dependence on individuals. Many organisations run on one colleague who knows how the exports work. A platform captures that knowledge in agreements and processes instead of in one head.

Interoperability according to DAMA: more than ETL

In the DAMA wheel, the reference model for data management, this knowledge area is called Data Integration & Interoperability. Those two words are chosen deliberately. Integration is moving data from A to B. Interoperability is systems also understanding each other: that ‘order date’ means the same thing in one system as in the other.

Most people know one form: ETL. Every night, data is extracted from the source systems, transformed into a shared format and loaded into the data warehouse. That is the lorry that runs once a day. Fine for management reporting and the month-end close, insufficient for many other situations. DAMA therefore counts several patterns under this area:

• Batch (ETL and ELT). The lorry on a fixed schedule. Reliable and cheap, but the data is usually hours old.
• Streaming and Change Data Capture (CDC). The conveyor belt. Every change in a source system is passed on immediately. Needed for fraud detection, stock management and customer service, where yesterday’s answer is too late.
• Application integration via APIs. The counter. One system asks another system for something and gets an answer there and then. The webshop asks the ERP: is this item in stock?
• Events and messaging. The bell that rings when something changes. A system announces ‘order placed’, and every system interested in that (invoicing, logistics, the data platform) acts on it itself. Sender and receiver do not need to know each other.
• Virtualisation and federation. The window. You look into another system without moving the data. Useful when copying is not allowed or not worth it.
• EDI (Electronic Data Interchange). The standard envelope between companies. Orders, delivery notes and invoices travel in a fixed message format (EDIFACT, X12, and for e-invoicing increasingly UBL via Peppol) between you and your suppliers or customers. Rarely visible, but in retail, logistics and manufacturing it is the backbone of the supply chain.
• File exchange and replication. Still common: a supplier delivers a file, or a database is mirrored one to one.

None of these patterns is better than the others. They suit different questions: how fast the data has to be there, how much of it there is and who takes the initiative. A mature platform usually runs four or five of them side by side.

Why application integration is a good place to start

Most data problems do not arise in the data warehouse, but at the moment two applications talk to each other. The webshop sends an order to the ERP. The CRM passes an address change to invoicing. It is at exactly those moments that it gets decided what a ‘customer’, an ‘order’ or a ‘product’ is and in what shape that information travels.

Whoever standardises those exchanges solves the problem at the source. One agreed order message with unambiguous fields means that invoicing, logistics and the data platform all receive the same thing. The alternative, letting every system send its own variant and having the data warehouse tidy up afterwards, works for a while and gets more expensive every year. Compare it to a distribution centre that asks suppliers for standard pallets instead of repacking every delivery itself. Between companies that idea has been normal for decades: EDI only works because both parties agree on the same message format before a single order is sent. Within the organisation itself, that discipline is often skipped.

Application integration is therefore the first layer of the data platform. The definitions and agreements you make there are inherited by the rest of the platform.

Architecture principles that make the difference

Not technical choices, but agreements your organisation records and enforces.

One definition per term. Before a system passes on data, the meaning of the terms has been agreed. ‘Active customer’, ‘net revenue’ and ‘delivery date’ have one definition, recorded in a business glossary, with an owner who is allowed to change it.

Loose coupling. Systems talk to each other through agreed interfaces, not by looking into each other’s databases. If you replace the CRM, only the connection needs adjusting, not every system that uses customer data.

Fix at the source, not downstream. A wrong address is corrected in the system where it was entered, not in twelve reports that use it.

The pattern follows the question. Choose batch, streaming, API or events based on the pace the process demands. Making everything real-time is expensive and rarely necessary; processing everything overnight makes fast processes impossible.

Every data flow has an owner. One person, not a department, who knows what passes through it, who depends on it and what happens if it stops.

Provenance is visible. For every figure on the platform it is possible to trace which source it came from and which transformations it has been through. That visibility is what keeps trust in the platform intact, and the auditor benefits along the way.

Build for reuse. A data flow is built once and used by several consumers. That is also the idea behind data products: one delivery with an owner and a contract, instead of a new export per question.

Where to start

Not with a platform selection and not with a three-year programme. Pick one process where the pain is visible: a figure two departments argue about, or a manual export that comes back every week. Map which systems are involved and where the definitions diverge. Agree one definition per term, standardise that single exchange between the applications and connect it to the platform. Measure what it delivers: how many hours of manual work disappear, how many arguments.

The second process goes faster. From the third onwards the approach becomes a pattern, and at that point a data platform has emerged without anyone having announced it as such.

Read more about how cimt approaches this under Data Integration & Streaming and Data Architecture & Lakehouse. Want to know where your organisation stands in this area? The DAMA maturity assessment gives a first picture of your position on integration and the other knowledge areas in half an hour. Or discuss your situation directly with one of our consultants.