How to prove a rebuilt system does exactly what the old one did

When an organisation replaces a system it cannot switch off, one question decides whether the move succeeds: does the new system do exactly what the old one did? The same interest on the same balance, the same rejected transaction, the same line on the same report. Proving that is called functional equivalence testing, and it is where most of the time and risk in a legacy modernisation goes.

A widely shared calculator for moving the world’s COBOL estimated 844 years of work. Testing, parallel running and acceptance made up 363 of them. We wrote about where those years go.

Every supplier will tell you their rebuild behaves the same. This guide is about how to tell whether they have proved it, and it works for any supplier, including us. Syntaxe makes Sovereign, which moves organisations off old software they cannot switch off and proves the new system behaves the same before anything is switched over.

Passing the tests is not the same as proof

Tests check what someone thought to test. The trouble with an old system is the behaviour nobody wrote down: a rounding rule added in 1994, a date check that only matters at year end, a field that quietly holds two meanings. Those are exactly the things a test plan written from today’s documents will miss, and exactly the things that cause the incident after go-live.

Real proof does not start from what people believe the system does. It starts from what the old system actually does, and compares the new one against it.

Matching the old system does not mean copying its mistakes blindly. Some differences turn out to be faults in the old system. A good proof shows each one, so that you choose whether to keep it or fix it.

Five questions to ask about any equivalence test

1. What exactly was compared? Ask for the list: printed output, every file written, every record in every file, every screen. “The results matched” means little until you know which results. The strongest proof compares everything the program produces, byte for byte.
2. Against what? The best reference is the old system itself, run on the same inputs. A comparison against a specification, a set of expected answers or a sample of production results is weaker, because each of those was written by a person and carries their assumptions.
3. On which inputs? Real data shows the common cases. It rarely shows the edges: the largest balance a field can hold, a zero, a negative, a leap day, an empty file. Ask whether the comparison also ran on generated data built to reach those edges.
4. Can you check it yourself? If only the supplier can repeat the proof, you are trusting the supplier, not the proof. Ask for the inputs, the outputs and a way to run the comparison again with your own team, without the supplier in the room.
5. What is not covered, and is it written down? Every proof has a boundary. A good one states it: which programs, which data, what was changed to make the old system run, what comes next. A proof that claims to cover everything usually covers less than it says.

Take these five questions to your next supplier meeting.

Three ways organisations prove a move today

A parallel run in production. The old and new systems process the same live work side by side for weeks or months, and people compare the results. It is realistic, but slow and costly, it only tests the work that happens to arrive in that period, and differences are often found by hand.

Comparing in the supplier’s environment. The supplier runs both systems in its own environment, often its cloud, and reports the comparison. This can be thorough. Ask whether your code and data have to leave your building, and whether you keep the evidence afterwards.

Comparing on your site, with proof you keep. The old and new run side by side inside your own network, on real and generated inputs, and every difference is shown with the inputs that caused it and the lines of old code behind it. You keep the evidence and can run it again. This is how Sovereign works.

These approaches work together. Comparing on your site finds the differences early, before any live work moves. A shorter parallel run then confirms the result on live work, and each part of the system switches over in steps, with an approval at each one.

What good evidence looks like

We rebuilt a public mainframe credit card application, CardDemo, and compared all 13 of its batch programs with the originals: every printed line and every written record, byte for byte, on its own data and on three sets of generated data. Every program matched, and every change needed to run the original off the mainframe is written down with the proof.

The first step

If you are planning a move off a system you cannot switch off, the first step with us is an assessment. Sovereign reads your code and gives you a report: what the system does, where the risk is, and what moving off it would take. The scope is fixed and agreed before any work starts, and you decide afterwards whether to go further.

Get in touch