Graphistania 2.0 - Episode 7 - The one after the Covid-19 lockdown

Yes! We were able to record and publish another episode of our Graphistania podcast. It's been an amazing and turbulent couple of months - but before the summer holiday season really takes off we wanted to get this to you.

Wishing you a fantastic and relaxing time - and in the mean time enjoy this episode!

Graphistania · Graphistania 2.0 - episode 7

Here's the transcript of our conversation:

What Recommender Systems and Contact Tracing have in common

With the Covid-19 pandemic raging in the past few months, I have had a lot of interesting conversations about the use of graph technology and how it could help the world be a better, safer, healthier place. At Neo4j, we even put in place a specific Graphs4Good program, helping out where we can. There's splendid research going on at Covidgraph.org, companies like Elsevier chipping in (and using Neo4j) as well, and I have tried to write up my humble thoughts on how Contact Tracing could really benefit from using graphs as well. See some of my recent posts published on this blog.

Looking at that work, however, I always had a the feeling that I was looking at an excellent example of something else: an excellent example of a great "graph problem". The contact tracing example is a great fit for a tool like Neo4j, and the reason why that is the case is basically because the problem that we are trying to solve with contact tracing (understanding the pandemic spread in our societies, predicting potential evolutions of the pandemic based on contacts between healthy and sick individuals, protecting the healthcare systems by managing the rate of spreading this way) is very much suited for analysis with graph technology. It is a domain where the links between people, the links between people and places, their visits, their meetings are the main important data entities that we need to look at. It's the connections that matter. It's the connections that are becoming the "equal citizens" in the dataset - and therefore we need to spend time and resources analysing it.

But of course I know one thing for sure: there are plenty of other cases that are like that, that are true "graph problems" and that could really benefit from a graph approach to solving it. We know that from all the Neo4j project that we have been running for years. So how do I demonstrate that? How do I show that Contact Tracing is essentially the same thing like a recommendation engine? Or another graph application that we have come to know and love. Let's explore that.

ESCO database in Neo4j: Skills, Competencies, Qualifications and Occupations form a beautiful graph!

Just a few weeks ago, I was discussing with Neo4j users that are active in the domain of "labour", or work. While talking to these users, they mentioned that there are standards out there that classify different types of work into different buckets (a taxonomy, if you will), and that there are two competing standards to do so out there. There's 

• the ESCO standard: the European Skills, Competences, Qualifications and Occupations, and 
• the ROME standard: the "Répertoire opérationnel des métiers et des emplois (ROME)"

The ESCO seems to be promoted by the European Commission, and the latter seems to be a Belgian/French initiative of some sorts. Surely they overlap, but I am not sure by how much. As luck would have it I started looking at the ESCO material first, but I am sure we could have written this post about ROME as well. It's the principles that matter.

And in principle, I figured that using these standards would be a really cool thing to do in Neo4j. Skills/Competences and  Occupations form really interesting graphy structures, and I could see how you could use a taxonomy like that to do some really interesting recommendations and other data workloads. So I wanted to give it a poke around.

Loading ESCO into Neo4j

The entire ESCO dataset can be downloaded from the European Commission's portal site: https://ec.europa.eu/esco/portal.  

It's really easy: you just select the data that you are interested in - the topic, format, and the languages - and put together a download package. 

In terms of format, you can choose between

• an RDF format, which basically gives you a large (500MB) Turtle file. Turtle - the Terse RDF Triple Language, see https://www.w3.org/TR/turtle/ - is probably more comprehensive, as it contains everything. But it's also quite a bit more difficult to manipulate and get your head around. I was able to import the Turtle file really easily using Jesus' "neosemantics" plugins, and had it up and running in minutes. But I found it more difficult to use - most likely because I am not an RDF afficionado. Sorry.
• CSV format. That's easy enough - we know how to import those. So all I needed to do was write a few Cypher scripts and import the data in a few minutes. I will put the scripts below, but you can also see them on github.

In any case, I opted to continue with the CSV files, and spent a little time importing the different files and connecting them together - in different languages. There's basically 5 files:

1. the Skills
2. the Skillsgroups, grouping the above together in groups
3. the Occupations
4. the ISCOgroups: this is a standard of the International Labour Organisation (ILO) that provides an International Standard Classification of Occupations. 
5. and then a few files with relationships between Skills and Occupations, different ISCO groups, and different Skills/Skillsgroups.

I wrote the script pretty quickly - it's really not that hard - and then I ...

... ended up with a few Neo4j databases:

1. one full of RDF triples - complicated!
2. one with English Skills, Skillsgroups, Occupations and ISCOgroups. 
3. one with Dutch Skills, Skillsgroups, Occupations and ISCOgroups.

In the Neo4j Desktop that looks a bit like this:

This is where the scripts are on Github.

Working with the ESCO database in Neo4j

Now that all that is imported, we can take a look at it. Let's start by looking at the model that we have imported. Pretty straightforward:

We can also just start looking at some data by just visually exploring it in the Neo4j Browser:

But it get's a lot interesting when we put Cypher to it, and start querying the data. For example, let me grab these two nodes here:

And look at the paths between them:

As always, the things that are located on the path, tend to be pretty interesting. Even more so when I think a bit more about the data, and start looking for the ESSENTIAL FOR relationship links. Let's see what comes back when I look for the links between a "software developer" and a "beer sommelier", when I ONLY traverse the relationships that define really important / ESSENTIAL relationships between Skills and Occupations:

Interesting. I am sure that a domain expert could do lots of other things here, especially if we could give that expert some non-technical tool like Neo4j Bloom.

All in all, this was a really easy and interesting experiment. I am sure there's a lot more to do here - but this was yet another example of a cool application of Neo4j in a surprising domain.

Hope this was useful.

Cheers

Rik

Using graphs for recommendations

Last month the wonderful people of Data Science Brussels invited me to do a talk about HR Analytics and how graphs fit into that. The material is over here. It was a great night, so when Philippe told me that the next meeting was going to be about Data Science in Marketing - I just had to invite myself. So yesterday, I had a the pleasure - again :) - to do a talk at the  about how you could use graphs in Marketing.

The thing that I focussed on was the topic of recommendations and recommender systems. In a world where content is king, and consumers are always looking for specific deals that would fit their profile best, these systems are going to become critically important.

In my mind, these systems always consist of two parts:

• a Pattern discovery system, where you - somehow - figure out what the patterns are that you are looking for. Could be by asking a domain expert, but could just as well be through some advanced machine learning algorithm. 
• a Pattern application system, where you start operationalising the patterns that you have found and applying them in real world applications, either in batch, or in near-real-time.

I believe that graphs and graph databases offer tremendous potential for this domain, and have tried to illustrate that in this talk. The key points being that

• Graphs excel at making these recommendations in real time. No more batch precalculations - just deliver the patterns you uncover as they develop.
• Graphs benefit from some of the most fascinating data analytics techniques out there: triadic closures, centrality measures, betweenness measures, PageRank scoring - all of which help you determine what parts of the graph really matter, or not.
• Graphs benefit from some operational advantages (among which: graph locality in your queries) to make them operationally very efficient.

I am sure I am missing stuff here, but these seem to be the most important pieces, to me.

The slides are on SlideShare, and included below:

The demonstration that goes with this was also recorded below. It's a little long, but I hope it's still interesting:

I hope this was a useful illustration for you of this fascinating use case. Any feedback - as always - greatly appreciated.

Cheers

Rik