A person reviewing an import needs to decide whether a new record belongs with a subject already in the system. Two names look alike. A date is close. Another record carries a stronger identifier but disagrees on a descriptive field. The interface offers a likely match, and someone has to decide whether to accept it.
A mistaken link can attach a result to the wrong subject, create a false relationship, or make separate observations appear to form one history. The person relying on that history may never see the original uncertainty. Correcting the join then means finding and explaining the views that depended on it.
This is identity resolution: deciding which records describe the same entity. Similarity can help find candidates. A useful product also lets someone inspect the evidence, leave an ambiguous link unresolved, and correct a decision without losing the original source records.
Define what is being identified
Start with the entity the application is trying to recognize. A subject, a sample, a source record, and an identifier assignment are different objects. Several samples can belong to one subject. Several source records can describe the same event. The same identifier string can occur in different namespaces.
Those distinctions determine which matching assumptions are legitimate. A rule requiring one source record to match at most one record in another file may be appropriate for a particular reconciliation. It may be inappropriate when reconciling multiple samples from the same subject.
Long-lived identifiers introduce another question: what does an identifier's assignment mean over time? A source may correct a transcription, retire an identifier, or reuse a local number in a new context. A mapping that ignores the issuing organization and the relevant assignment period can create a confident-looking but unsupported join.
The domain definition should therefore precede algorithm selection. Write down the entities, identifier namespaces, allowed cardinalities, and correction responsibilities. A similarity method can then operate inside a defined task.
Candidate generation is part of the evidence pipeline
Comparing every record with every possible counterpart can be expensive. Linkage systems commonly reduce the candidate set before calculating detailed comparisons. That restriction makes the work feasible, but it also determines what the system can ever find.
If a true counterpart is excluded at this stage, a better scoring model cannot recover it later. Candidate coverage and candidate scoring need separate evaluation.
The Office for National Statistics describes linkage as a multistage process and discusses tools for preparing candidate comparisons. Its methods paper also carries a useful correction: it recommends precision and recall rather than the accuracy formula removed from an earlier version. ONS linkage methods provide a concrete reminder to check both the method and its publication history.
In a product, candidate generation should leave enough information to investigate a miss. A reviewer may need to know which source was searched, which fields were available, and which restrictions excluded a record. “No match found” is stronger than the evidence if the actual result was “no candidate passed this search rule.”
A score, a probability, and a decision are different commitments
A string similarity score describes a comparison under a particular rule. A match weight can summarize evidence under model assumptions. A probability requires an interpretation tied to a model or calibration basis. A decision adds a policy about what action is justified.
These distinctions matter in the interface. A number displayed as 0.94 can look like a 94% probability even when it is simply a normalized similarity measure. Reviewers need to understand what the number actually represents and what information it omits.
Even a well-supported match probability does not settle the operational decision. The consequences of a false link may differ from the consequences of leaving two records separate. An application may require stronger evidence before a merge that affects published reports than before showing a candidate in a review queue.
That policy should be explicit. The algorithm supplies evidence; the application determines what disposition and authority follow from it.
Pairwise decisions can conflict with the task
Consider a fictional two-file reconciliation with a one-to-one rule. Record A1 has a strong accepted link to B1. Record A2 has plausible candidate links to both B1 and B2. A3 has no suitable candidate.
Independent pairwise decisions can assign both A1 and A2 to B1. Under this task's one-to-one assumption, the decisions conflict even if both pair scores look favorable in isolation.
Sadinle's methodological work models bipartite record linkage as a constrained matching and develops partial estimates that can leave uncertain assignments unresolved. The author manuscript supports the importance of representing uncertainty at the matching level, rather than treating every pair as an unrelated classification.
The product lesson is to record the assumptions that make an assignment valid. A constraint can resolve a conflict under one task and manufacture a mistake under another. It should be visible in the review context and in the evidence retained with the decision.
Unresolved identity can be a useful result
A system that must select a winner for every record can hide uncertainty in its data model. The interface may look complete while downstream calculations depend on arbitrary tie-breaking.
An unresolved outcome creates a different product responsibility. The application should explain what remains ambiguous, which alternatives are plausible, and what evidence could settle the matter. It should also identify what work may continue while the link remains open.
For example, a source document can be retained and its fields reviewed without merging its subject into a canonical record. A result can remain attached to the source identity while the identity question is investigated. A downstream relationship can be marked conditional on a proposed link rather than displayed as an accepted fact.
This does not require presenting every possible world to every user. It requires keeping uncertainty available where it can change a decision. A qualified reviewer should be able to inspect why the system stopped and act on a concrete question.
Carry consequential uncertainty into derived views
Suppose a proposed link determines whether two observations belong to the same subject. Accepting the link may create a trend; rejecting it may leave two separate histories. The uncertainty is now relevant to the trend's interpretation.
The application can examine how much a downstream conclusion depends on that link. It can calculate a view under each plausible assignment or flag the specific dependency for review. Such an exercise establishes sensitivity to an assumption. It does not prove that any candidate assignment is the true identity.
This distinction matters in pedigree, longitudinal, and research products. A graph algorithm can produce a precise result from a supplied graph while the graph's identity assumptions remain unresolved. Precision in the calculation should not erase uncertainty in the inputs.
A useful explanation identifies the dependency: the result changes if link L is accepted, and the evidence for L is still under review. That gives a person a way to prioritize the identity work that matters most.
Corrections should preserve the original source account
An accepted identity assertion may later be corrected. The correction is easier to manage when the system retained the source records and the history of the link.
A reversible design can preserve which records were joined, the reason, the actor or process that accepted the link, and the outputs that relied on it. Correcting the link can then trigger a review or rebuild of affected views without changing what the original sources actually supplied.
A destructive merge that discards source distinctions makes this harder. The team may be left reconstructing why a field exists or which subject a historical report originally described.
The same design helps ordinary operations. A support person can explain why two records are connected, a reviewer can compare alternatives, and an engineer can identify which derived outputs need attention after a correction.
Evaluate the join and its consequences
A high overall match rate says little about the quality of the accepted links. A system can link many records incorrectly, or link only the easiest cases and leave a biased subset unresolved.
The size of the candidate universe can also flatter an ordinary pair-classification metric. Suppose two fictional files each contain 1,000 records, with exactly 1,000 true one-to-one links among the million possible pairs. A classifier that rejects every pair is correct on 999,000 pairs under that binary definition of accuracy. Its pair accuracy is 99.9%, while its recall is zero. Precision is undefined because it accepts no links.
That constructed example is separate from the formula withdrawn by ONS. It shows why the unit and denominator of a reported metric need to be explicit. A reviewer needs to know whether the evaluation covers all possible pairs, generated candidates, accepted links, or subjects successfully reconciled. Each denominator describes a different part of the pipeline.
Evaluation should consider false links, missed links, candidate coverage, and the distribution of unresolved cases. It should also ask whether errors concentrate in particular sources, identifier patterns, or populations relevant to the product.
The UK government's methodological discussion of linkage quality emphasizes that linkage errors can affect subsequent analysis and that the linker and analyst need to exchange relevant information. Quality assessment in data linkage supports treating the join as part of the analytical pipeline.
Human review needs evaluation too. A reviewed pair is not automatically error-free. A sampling design for checking accepted and rejected pairs should describe how cases were chosen and what reference information was available. Precision and recall estimates need that context to be interpreted.
For a first release, choose a bounded source pair and a meaningful downstream use. Demonstrate candidate generation, an accepted link, a conflict, an unresolved record, and a correction that updates the affected view. The useful deliverable is a reconciliation process the team can inspect and improve, with enough history to explain why the records are connected.
Protabyte builds data-intensive products with explicit record identity, source context, and review paths. Talk with us about an import or reconciliation workflow your team needs to trust.