DadjokeGraph Part 4/6: Adding NLP and Entity Extraction to prepare for further disambiguation

A Graph Database and a Dadjoke walk into a bar...

As we can see in the pyjamazon example from before, the disambiguation of our Dadjokes has come a long way - but is not yet complete. Hence we we call the graph to the rescue here, and take it a final step further that will provide a wonderfully powerful example of how and why graphs are so good at analysing the structural characteristics of data, and make interesting and amazing recommendations on the back of that.

Here's what we are going to do:

1. we are going to use Natural Language Processing to extract the entities that are mentioned in our Dadjokes. Do do that, we are going to use the amazing Google Cloud NLP Service, and call it from APOC. This will yield a connected structure that will tell us exactly which entities are mentioned in every joke.
2. then we are going to use that graph of dadjokes connected to entities to figure out if the structure of the links can help us with further disambiguation of the jokes.

So let's start with the start.

Making sense of the news with Neo4j, APOC and Google Cloud NLP

Recently I was talking to one of our clients who was looking to put in place a knowledge graph for their organisation. They were specifically interested in better monitoring and making sense of the industry news for their organisation. There's a ton of solutions to this problem, and some of them seem like a really simple and out of the box toolset that you could just implement by giving them your credit card details - off the shelf stuff. No doubt, that could be an interesting approach, but I wanted to demonstrate to them that it could be really much more interesting to build something - on top of Neo4j. I figured that it really could not be too hard to create something meaningful and interesting - and whipped out my cypher skills and started cracking to see what I could do. Let me take you through that.

The idea and setup

I wanted to find an easy way to aggregate data from a particular company or topic, and import that into Neo4j. Sounds easy enough, and there are actually a ton of commercial providers out there that can help with that. I ended up looking at Eventregistry.org, a very simple tool - that includes some out of the box graphyness, actually - that allows me to search for news articles and events on a particular topic.

So I went ahead and created a search phrase for specific article topics (in this case "Database", "NoSQL", and "Neo4j") on the Eventregistry site, and got a huge number of articles (46k!) back. 

Exploring the Congo Holdup articles with Neo4j

As you may know, I am from Belgium. I love this little queer country, with all of its idiosyncracies and weirdness it still makes for a great place to live. Did I mention Beer, Waffles and Chocolate yet? Yes, that's why.

But in the 191 years that we have existed as a country, we have done some weird sh!t as well. As an example, we did some of the craziest stuff ever under our former King Leopold II. I got to know a bit more about that through Klara, a (Dutch spoken, but nevertheless crazy wonderful) Flemish radio station, which hosted a podcast about Leopold II's crazy, and sometimes cruel, adventures in Congo. See over here if you are interested. It's a terrible, but fascinating story.

So more recently, when I started reading about the Congo Holdup in De Standaard, the newspaper that I subscribe to over here, I could not be but interested - and wanted to know more. I have been reading about it on an off, and have made my way to the actual source of this investigation, which is run by the EIC - the European Investigative Collaborations - you can find the details over here. Essentially, this is a about How to rob a country, using a bank. Specifically, this is the BGFIBank, which laundered lots of unsavoury transactions through its books - and allowed some of the key leaders of Congo to get away with crazy big amounts of $$$ - leaving the country's citizens in poverty in their wake.

So this post is NOT going to be about the details of that investigation. It's more about how you can use a graph, and some of the easiest tools ever, to better make sense of a pile of articles like that. That's what I have tried to do, and what I would like to highlight below. I will leave the actual interpretation of this data to others.

Cognitive biases in Neo4j

I am an economist/engineer. I studied "Commercial Engineering" in Belgium in the nineties, and was quite an avid learner of economic theories large and small at the time. I did however, always kind of find myself uneasy at economists insistence on the rationality the homo economicus, as I knew, and observed all around me, that people were far from rational. That's why, ever since I learned of its existence, I have been a big fan of the field of behavioral economics - which actually tries to formulate ecomic theories that are real, and often times, irrational. I fondly remember first reading Dan Ariely's Predictably Irrational, and learning about some of the crazy biases that he observed and DESCribed. And Nobel-prize-winning Daniel Kahneman has been a hero for decades. I think about the Framing Effect) and Prospect Theory almost on a daily basis.

It all started with a tweet

So you can imagine my excitement when I learned about this tweet:

Should be taught to all at a young age pic.twitter.com/GlVkjcdhah

— Elon Musk (@elonmusk) December 19, 2021