Duplicates in data is always an issue. An issue which is magnified once you start trying to load said data into a visual analysis tool like i2. The nature of tools like i2 is that they find connections between entities; in order for that to work the data must be clean a free of duplicates.

In reality, it’s not always feasible or realistic to get rid of all duplicates. For this reason, the i2 analysis suite does provide some matching functionality

Smart Matching – i2 Analyst’s Notebook

To explain it in a simple way: Smart matching in analyst’s notebook is achieved by categorising entities and applying almost human matching logic.

For example, a visualisation may contain a police officer, a victim, a suspect, a prisoner, a male and a female entity. These entities could all be categorised as People. As humans, we could look at a list of people and would instinctively apply matching logic. We’d know that date of birth was important. As are surnames, although they may change when someone marries. We know that Chris could be spelt in several different ways and (as with any data) there may be spelling mistakes.

i2 smart matching can do the same thing. It will apply a different set of logic to different categories of entities to give an almost human matching ability.

Smart matching example

Great! Problem solved! Well yes, if you have a smallish set of data then smart matching will work well. However, for smart matching to work, the entities need to be on an analyst’s notebook chart. This may start becoming an issue once you hit the 20,000 entities mark! You could try some kind of batching process but it’s likely to be time-consuming and not entirely accurate.

Matching in iBase

Matching in iBase example

iBase will not have this problem. Being a database it will have access to all of your records without needing to render them on a visual chart. The problem we have in iBase is it doesn’t have access to the same smart matching logic. With iBase we can match someone with exactly the same surname. Or, the exact same surname and date of birth. While useful, this will miss a lot of duplicates and therefore needs to be used with care.

Custom Matching Functionality

For some of our clients, the functionality described above didn’t meet their needs. They either had too many entities or wanted to find duplicates of completely different entity types, or maybe over different databases.

For these clients s-branch developed an external matching solution, using an existing string matching theory called Levenshtein distance. Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other. What’s great about using Levenshtein distance is that it gives us a score. Meaning we can see narrow down our results to only the most relevant matches.

Custom matching example

The external matching is computed outside of iBase. This is done to minimalise the effect on database performance but also allows us to iterate through entire datasets programmatically. This means we can compare duplicates regardless of whether they are the same entity type; or even the in the same database. The output from this is a matching score which is then imported into iBase as a ‘AutoMatch’ link. Our clients can then use existing iBase functionality, such as queries and sets to review the matches found and if applicable merge them through the normal UI.

You can read more on Levenshtein distance here.  To find out more about using Levenshtein distance with iBase or to discuss your own duplicate problem, please contact us

In October 2017 I was asked by Arquebus Solutions to speak at the IFFS, an event designed for individuals who investigate firearms-related crimes, heads of police agencies and those who shape policy aimed at combating firearm crime.

It was an honour and a privilege to speak at such an event. It was also fascinating and humbling to hear experts share lessons learned following the recent terrorist attacks in Europe and other parts of the world.

Firearm Focal Points

One point which was consistent throughout the event was Firearm Focal Points (FFPs). FFPs are designed to collect all the necessary firearm information together, creating accurate investigative leads thereby helping both investigators and prosecutors to tackle the problem at source.

Firearm Focal Point

FFPs work because of data collaboration. Analysis of ballistic evidence has advanced hugely but this information should not be analysed on its own. If a cartridge case is found at a location what other crimes have we seen in that area? Were there any ‘shots fired’ call outs at that time? We may know that the cartridge case comes from a particular type of firearm. What intelligence do we have about trafficking those types of firearms? If we are lucky enough to find or have a match to a particular firearm; who is it registered to? Is there any link to other crimes?

We can only answer these questions if we have access to all the information. Analysis tools allow us to understand this information quickly, helping FFP’s provide accurate leads fast. It was my job to showcase how analysis tools can aid investigations when this data is made available.

Arquebus and I worked together to implement the Kosovo FFP. The success of this deployment was collaborated by Kosovo Police, who also spoke at the conference. It was with great pride that I listened to real cases where the collaboration of ballistic and crime information had aided investigations.

Crime in Action

A unique feature of IFFS 2017 was the ‘Crime in Action’ simulation, which ran through the entire conference. Subject experts shared their experiences and expertise not only through presentations and interactive panel discussions but also as part of a ‘crime in action’ scenario.

IFFS

It was great to see all of the skills and expertise employed in dealing with a live gun crime investigation, but also to see how areas such a legislation, policy making, joint working arrangements and international development all play a part in how we deal with gun crime and illicit arms trafficking.

The highlights for me were visiting a simulated gun crime scene and the final arrest which occurred in the middle of the conference, much to the surprise and delight of the delegates.

In summary, it was one of the best conferences I’ve attended and I hope to return either as a delegate or presenter in the future.

Introduction

Large amounts of text can pose a challenge to analyse and understand. We keep generating more of it every day. This ‘unstructured text’ can be in documents, emails or social media posts. Indexing this text can help when searching for keywords, but what if you don’t know the keywords? What if you don’t know where to start?

This post describes some of the technology and techniques that are available. It focusses on free to use (but not to distribute) software. It will not focus on particular commercial software; although some products are mentioned.

The Data

Case Study 21

The data I’ve used for this is real. A case study from a public inquiry into Satyananda Yoga Ashram at Mangrove Mounting (Australia) for allegations of child sexual abuse. The allegations are made against a spiritual leader Akhandananda in the 1970’s and 1980’s with submissions from survivors, held in multiple documents.

https://www.childabuseroyalcommission.gov.au/case-studies/case-study-21-satyananda-yoga-ashram

WARNING: The content of these documents are disturbing. Reader discretion is advised. 

Natural Language Processing
To help understand the data within these documents, I’ve first used Natural Language Processing (NLP). NLP allows machines to ‘read’ unstructured text. One of the ways NLP evaluates a document is: Named Entity Recognition (NER). NER gives an understanding of what entities (for example: people, organisations or locations) are present in the text.

Natural Language Processing (NLP)

To demonstrate NLP I’ve used free software created by Stanford University. There are many other good commercial NLP products on the market offering various language/analytic capabilities.

Statistical Analysis

To give meaning to the results of the NLP I’ve summarised the documents and their contents dependant on the entity type. In this example, I’ve stuck with people, organisations and locations. Below is a statistical visualisation.

Statistical – PowerBI

For each entity type, a count of occurrences is shown as well as document coverage. ‘Akhandananda’ is the person that has appeared the most times and the document ‘Transcript – Day 108’ contains the most people.

This basic visualisation was built very quickly using the free version of PowerBI. The free version allows you to create visualisations but limits the sharing options available. There are other good tools available for this kind of analysis, most offer free trials which is great if you want to compare functionality.

Visual Analysis

To take the analysis further I really needed to load the results into a database. I chose a graph database, simply because graph databases perform well with entity data such as people, organisations and locations.

For this example, I’ve loaded my data into Neo4j Community Edition which is free to use (under the GPL v3 license). If I was building something commercial which needed to scale then Neo4J offer commercial licenses as well.

Once modelled, I could easily generate networks uncovering interconnecting entities within the data. The resulting visualisation is based on a short query. People are represented in green, organisations in blue and locations in pink.

Inquiry Network

Here we can see that Akhandananda and Shishy are key entities, mentioned alongside a lot of other entities. We can also see that Commission, Satyananda, Tim Clark, DWYER etc are linked to both Akhandananda and Shishy. These now become entities of interest.

This gives a place to start. I first concentrated on Satyananda. Satyananda is a complicated entity as it’s the name of an individual and also the Yoga organisation. In this case, NLP has pulled out Satyananda the person. I used Neo4J to drill down into these entities, showing three documents that all entities were linked to. This time the numbers on the links refer to the occurrences of that entity in that document.

Document Network

A quick look through those three documents soon shows Satyananda, Akhandananda and Shishy being mentioned together multiple times. The subject matter is disturbing, so I’ve chosen the text carefully. Below is a screenshot where Shishy is mentioning both Satyananda (as a person) and Akhandananda when being questioned.

Highlighted Text

Summary
Using a mixture of NLP, statistical and visual analysis tools allowed me to very quickly narrow down which documents and entities are of interest.

In truth, with a small dataset like this (under 40 documents) reading each document manually is another valid option. NLP is also not completely accurate and cannot fully replace the human touch. If faced with 1000 + documents then techniques like above can really help direct analysts to the key information.

This is a demonstration only and focusses on freely available tools. There are many commercial offerings that offer the functionality described above (and more) in one seamless application. If you are looking for fraud or crime in unstructured text then S-branch can help guide you to the appropriate products available.

i2
Since 1990 i2 has been providing its clients with accessible and intuitive data analysis tools. Analyst’s Notebook has now become the standard for charting data and is used in nearly all of the UK police forces. The potential of Analyst’s Notebook sitting on top of a database was very quickly understood. This understanding spawned the release of iBase, iBridge and later on enterprise solutions such as IAP.

This movement towards visualising larger datasets made the offering applicable to not only law enforcement but to government agencies, military, and commercial clients as well. If you have a sporadic flow of data in formats that vary greatly, i2 is still one of the most compelling tools on the market.

If you have a sporadic flow of data in formats that vary greatly, i2 is still one of the most compelling tools on the market.

S-branch i2 Consultancy
We’ve been part of the i2 journey since 2006, either as trainers, technical consultants or marketers. Through these roles, we’ve worked with several police forces both in the UK and abroad, charities, insurance firms, law firms, government agencies and commercial clients.

Here are some of the ways S-branch can assist i2 users:

• Schema Design – By far the most important part of any i2 deployment. Understanding the data landscape and the client’s needs are more important than technology. S-branch has many years experience with thinking in ‘Entity and Links’ and can apply this to new and existing i2 deployments
• Implementation – i2 has a straightforward deployment model which fits well with most IT infrastructures. The integration with other data sources or systems is the technical part. S-branch has experience with working with various I.T departments and third-party providers to assist with this
• Health Check – Like anything complicated, an i2 deployment could do with a check-up every now and then. If a client has had their deployment for a while or is worried about any aspect of its setup, S-branch can assist with this.
• Enhancements – i2 does a lot of things very well but no tool does everything. The idea of moving to a different platform just to get that one piece of functionality is daunting. S-branch has experience in enhancing the product set, to ensure the client gets the best out of their existing investment. This has ranged from: data washing, alerting, advanced duplicate detection and even advanced pre-processing to get more out of their data.
• i2 Administration – i2 is often a ‘mission critical’ system for a lot of our clients. Having an administrator that knows the software inside and out, as well as the underlying technology is a reassurance. To these clients, S-branch can be that semi-regular resource that can be called upon.

If any of the above could benefit your i2 deployment or you’re looking into data analysis tools in general; please feel free to contact us.

Following an initial period of successful freelancing, I’m delighted to announce that the S-branch brand has now been set up.

I’d like to thank all of our existing customers for their support so far and will ensure that the transition is a painless one.

S-branch will continue to assist it’s clients in identifying crime and fraud in large amounts of data. While experienced with individual pieces of software, S-branch will remain software agnostic ensuring our clients get the best solution for their needs.