Duplicates in data is always an issue. An issue which is magnified once you start trying to load said data into a visual analysis tool like i2. The nature of tools like i2 is that they find connections between entities; in order for that to work the data must be clean a free of duplicates.
In reality, it’s not always feasible or realistic to get rid of all duplicates. For this reason, the i2 analysis suite does provide some matching functionality
Smart Matching – i2 Analyst’s Notebook
To explain it in a simple way: Smart matching in analyst’s notebook is achieved by categorising entities and applying almost human matching logic.
For example, a visualisation may contain a police officer, a victim, a suspect, a prisoner, a male and a female entity. These entities could all be categorised as People. As humans, we could look at a list of people and would instinctively apply matching logic. We’d know that date of birth was important. As are surnames, although they may change when someone marries. We know that Chris could be spelt in several different ways and (as with any data) there may be spelling mistakes.
i2 smart matching can do the same thing. It will apply a different set of logic to different categories of entities to give an almost human matching ability.

Smart matching example
Great! Problem solved! Well yes, if you have a smallish set of data then smart matching will work well. However, for smart matching to work, the entities need to be on an analyst’s notebook chart. This may start becoming an issue once you hit the 20,000 entities mark! You could try some kind of batching process but it’s likely to be time-consuming and not entirely accurate.
Matching in iBase

Matching in iBase example
iBase will not have this problem. Being a database it will have access to all of your records without needing to render them on a visual chart. The problem we have in iBase is it doesn’t have access to the same smart matching logic. With iBase we can match someone with exactly the same surname. Or, the exact same surname and date of birth. While useful, this will miss a lot of duplicates and therefore needs to be used with care.
Custom Matching Functionality
For some of our clients, the functionality described above didn’t meet their needs. They either had too many entities or wanted to find duplicates of completely different entity types, or maybe over different databases.
For these clients s-branch developed an external matching solution, using an existing string matching theory called Levenshtein distance. Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other. What’s great about using Levenshtein distance is that it gives us a score. Meaning we can see narrow down our results to only the most relevant matches.

Custom matching example
The external matching is computed outside of iBase. This is done to minimalise the effect on database performance but also allows us to iterate through entire datasets programmatically. This means we can compare duplicates regardless of whether they are the same entity type; or even the in the same database. The output from this is a matching score which is then imported into iBase as a ‘AutoMatch’ link. Our clients can then use existing iBase functionality, such as queries and sets to review the matches found and if applicable merge them through the normal UI.
You can read more on Levenshtein distance here. To find out more about using Levenshtein distance with iBase or to discuss your own duplicate problem, please contact us