ReBeerGraph: importing the Belgian BeerGraph straight from a Wikipedia HTML page

I have written about beer a few times, also on this blog. I have figured out a variety of ways to import The Wikipedia Page with all the belgian beers into Neo4j over the years. Starting with a spreadsheet based approach, then importing it from a Google Sheet (with it's great automated .csv export facilities), and then building on that functionality to automatically import the Wikipedia page into a spreadsheet using some funky IMPORTHTML() functions in Google sheets.

But: all of the above have started to crumble recently. The Wikipedia page, is actually kind of a difficult thing to parse automatically, as it splits up the dataset into many different HTML tables (which makes me need to import multiple datasets, really), and then it also seems like Wikipedia has added an additional column to it's data (the "Timeframe" or "Period" in which a beer had been brewn), which had lots of missing, and therefore empty cells. All of that messes up the IMPORTHTML() Google sheet that I had been using to automatically import the page into a gsheet.

So: I had been on the lookout for a different way of doing the automated import. And recently, while I was working on another side project (of course), I actually bumped into it. That's what I want to cover and demonstrate here: importing the data, directly from the page, without intermediate steps, automatically, using the apoc.load.html functionality.

Read more »

Part 1/3: Wikipedia Clickstream analysis with Neo4j - the data import

Alright, here's a project that has been a long time in the making. As you may know from reading this blog, I have had an interest, a fascination even, with all the wonderful use cases that the graph ecosystem holds. To me, it continues to be such a fantastic thing to be able to work in - graphs are everywhere, and more and more people are waking up to the fact that they really should look at their data as a network, and leverage the important relationships that are often hidden from plain sight.

One of these use cases that has been intriguing me for years, literally, is clickstream analysis. In fact, I wrote about this already back in 2013 - amazing when you think about it. Other people, like our friends at Snowplow Analytics, have been writing about this as well, but somehow the use case has been snowed under a little maybe. With this blogpost, I want to illustrate why I think that this particular use case - which is really a typical pathfinding application when you think about it, is such a great fit for Neo4j.

A real dataset: Wikipedia clickstream data

This crazy journey obviously started with finding a good dataset. There's quite a few of them around, but I wanted to find something realistic, representative and useful. So after some digging around I found the fantastic site of Wikimedia, where they actually structurally make all aggregated clickstream data of Wikipedia's pages available. You can just download them from this their website, and grab the latest zipped up files. In this blogpost, I worked with the February 2021 data, which you can find over here.

When you dowload that fine, you will find a tab-separated text file that includes the following 4 fields

• prev: the previous page that the navigation came from
• curr: the current page that the navigation came into
• type: the description of the type of navigation that was occuring. There's different possible values here
• link: a regular link between pages
• external: a link from an external page to the current page
• other: a different type - which can occur if people try to hide their navigation patterns
• n: the number of occurrences of the (prev, curr) pair - so the number of times this navigation took place.

So this is the dataset that we want to import into Neo4j. But - we need to do one tiny little fix: we need to escape the “ characters that are in the dataset. To do that, I just opened the file in a text editor (eg. TextEdit on OSX) and did a simple Find/Replace of " with "". This take care of it.Read more »

Part 2/3: Wikipedia Clickstream analysis with Neo4j - queries and exploration

In the previous blogpost, I showed you how easy it was to import data into Neo4j from the official Wikipedia clickstream data. I am sure you would agree that it was surprisingly easy to import a reasonably sized dataset like that, within a very reasonable timeframe. So now we can have some fun with that data, and start applying some graph queries to it. All of these queries are also on github, of course, and you can play around with them there as well.

So let's take a look at some of these queries. 

Some data profiling and exploration

Here's a very simple query to give you a feel for the dataset:

match (n)-[r:LINKS_TO]->(m)

return distinct r.type, count(r);

match (n) return count(n);

The results are telling:

And so now we can start taking a look at some specific links between pages. One place to investigate would be the Neo4j wikipedia page. Here's a query that looks at the source pages that are generating traffic into the Neo4j wikipedia page:

Read more »

Part 3/3 - Wikipedia Clickstream analysis with Neo4j - some Graph Data Science & Graph Exploration

In the previous blogposts, I have tried to show

• How easy it is to import the Wikipedia Clickstream data into Neo4j. You can find that post over here.
• How you can start doing some interesting querying on that data, with some very simple but powerful Cypher querying. You can find that post over here.

In this final blogpost I want to try to add two more things to the mix. First, I want to see if I can do some useful "Graph Data Science" on this dataset. I will be using the Neo4j Graph Data Science Library for this, as well as the Neuler Graph App that plugs into the Neo4j Desktop. Next, I will be exposing some of the results of these Graph Data Science calculations in Neo4j's interactive graph exploration tool, Neo4j Bloom. So let's do that. 

Installing/Running the Graph Data Science Library

Thanks to Neo4j's plugin architecture, and the Neo4j Desktop tool around that, it is now super easy to install and run the Graph Data Science Libary - it installs in a few clicks and that you are off to the races:

Read more »

Graphistania 2.0 - Episode 5 - This Month in Neo4j

Friends.

These are interesting times. These are difficult times, but we can deal with it together, as a community, as a graph. So that's why we were super happy that, just as Belgium was going into lockdown last week, we were able to record another Graphistania podcast episode for you, talking about the world in general, but also covering some of the amazing graph use cases that drifted over our screens in the past month, in the This Week in Neo4j (TWIN4J) newsletter.

There were actually many things to talk about, in terms of fascinating graph use cases, and I will highlight only the most striking ones here.

Our friends at Kineviz did some really interesting and timely work on  COVID-19 temporal and spatial data visualization. This stuff is really important to understand, as pandemic spreads clearly follow graph patterns. Read Connected if you are not convinced. 

Worth highlighting: Bloodhound: Windows network penetration testing with Neo4j, had a new release that you might want to take a look at. If you are not familiar with Bloodhound yet, you may also want to check out my interview with the Bloodhound crew on this podcast a while back. 

We published this fun little thing called a Neo4j Treasure Map - check it out! 

Finally - we also have a a Winegraph! It's a great example of importing data from the web using Norconex.  

Some interesting stuff on using Neo4j for Gene ID mapping: take a look! 

Another examle of enriching graphs with Wikidata, from the one and only Mark Needham: look at Mark's blog over here! 

Don't forget: we Introduced the Neo4j Graph Data Science plugin with examples from the "Graph Algorithms" book. 

A really interesting tweet about a visualisation of the US Supreme court as a graph db... Would love to see more like that. 

And for some fun: Pokégraph: Gotta Graph 'Em All! 

Some important stuff: we did a great 4.0 webinar that is giving you a lot of info on what to expect in the new version of Neo4j.  

There was a great update to NeoMap: Visualizing shortest paths with neomap ≥ 0.4.0 and the Neo4j Graph Data Science plugin.

Those were the most important ones. So let's talk about these now - I am sure there's a lot of cool stuff here fore everyone!
Read more »

Podcast Interview with Sascha Peukert, TU Dresden

Another week another podcast episode! Here's a great chat with a graphista that has not even started his professional career yet - but that, by the sounds of it - could really be a stellar one. My chat with Sascha Peukert in Germany:

Here's the transcript of our conversation:

RVB: 00:04.386 Hello, everyone. My name is Rik, Rik Van Bruggen from Neo Technology, and here we are recording another podcast session for our Graphistania podcast. And today I have a very interesting guest on the podcast from Germany - from Dresden. And that's Herr Sascha Peukert. Hi, Sascha. 

Read more »

Challenge ACCEPTED: Belgian TV holds a "wiki wiki" challenge

My friend and future colleague Pieter alerted me to an interesting challenge hosted  on Belgian Television: a Wiki Wiki Challenge. The assignment is simple: find the link between two pages on the Dutch Wikipedia. How difficult can it be? Well, pretty difficult if you look at the assignments:

1. Nicole Kidman -> TO -> Hugo Claus
2. De ring van Brussel -> TO -> Thuis
3. Okselhaar -> TO -> Postmodernisme
4. Henk Rijckaert -> TO -> Chuck Norris

That could be interesting. So what to do? Waste my time browsing through thousands of wikipedia pages? Or use a sharp tool, and use it wisely? Could that tool be called, Neo4j, by any chance?

Importing nl.wikipedia.org into Neo4j

In order to calculate these paths between these pages, we first would have to import the Wikipedia articles into Neo4j. We had done this once before, using Graphipedia, a very easy piece of software to convert the downloaded wikipedia archive into a Neo4j database. It generates a 2.0 store format, but that's an easy upgrade to the 2.2M04 that I have running on my machine. The import was done in less than 30mins on this machine. So then we could start playing around...

Just quickly browsing through the "Pages" (a label in this database) and the "Links" (a relationship type in this database) gave us a feel for the assignment within Neo4j.

All we needed to do was to grab the two nodes in the Wiki Wiki challenge assignments, and run a "ShortestPath" algorithm on it - which is conveniently part of the Cypher query language. That would be it - so let's try it out.

Answering the Challenge Questions

Let's go through the 4 questions:

1. Nicole Kidman -> TO -> Hugo Claus

The query for this is pretty easy: 

 match (hc:Page {title:"Hugo Claus"}), (nk:Page {title:"Nicole Kidman"}),   
 p = allshortestpaths((hc)<-[*]-(nk))  
 return p  

which gives you this result. So apparently there are quite a few paths possible:

If we just limit it to one result

you quickly figure out that we are going from

• http://nl.wikipedia.org/wiki/Nicole_Kidman, to
• http://nl.wikipedia.org/wiki/20_juni (Nicole's birthday), to
• http://nl.wikipedia.org/wiki/1971 (Audi Murphy died on 20th of June 1971), to
• http://nl.wikipedia.org/wiki/Hugo_Claus (he published his novel "Schola Nostra") in that year.

Job done!

Similarly we can do the same with the next assignment:

2. De ring van Brussel -> TO -> Thuis

Same kind of query

 match (t:Page {title:"Thuis (televisieserie)"}), (rb:Page {title:"R0 (België)"}),   
 p = allshortestpaths((t)<-[*]-(rb))  
 return p  

and the result is in the city of Dilbeek, where there is a horseriding stable called "Hof ter Smissen" that is often featured in the TV show Thuis.

Again: job done! Onto the third (bizarre) assignment!

3. Okselhaar -> TO -> Postmodernisme

What is the link between armpit hair and postpodernism? We always wanted to know. A quick query later, we see the answer:

 match (o:Page {title:"Okselhaar"}), (pm:Page {title:"Postmodernisme"}),   
 p = allshortestpaths((o)-[*]->(pm))  
 return p  

This answer is obviously a lot less trivial:

• start at http://nl.wikipedia.org/wiki/Okselhaar
• go to http://nl.wikipedia.org/wiki/1915 - the year when armpit shaving started in the Western World
• go to http://nl.wikipedia.org/wiki/1992, the year that Harry Prenen (who was born in 1915) died
• go to http://nl.wikipedia.org/wiki/Jeff_Koons, home page of the artist that created Puppy in 1992. 
• end up at http://nl.wikipedia.org/wiki/Postmodernisme, which is the style that Jeff Koons is categorized as.

Here you can see that result in the Neo4j browser:

And another one bites the dust! One more to go!

4. Henk Rijckaert -> TO -> Chuck Norris

Probably the most telling and funniest link is the one between Henk Rijckaert and Chuck Norris. We only need three hops:

• start at http://nl.wikipedia.org/wiki/Henk_Rijckaert
• go to http://nl.wikipedia.org/wiki/16_augustus, his birthday
• go to http://nl.wikipedia.org/wiki/1940, the year that Henri Desgrange died on the 16th of August.
• go to http://nl.wikipedia.org/wiki/Chuck_Norris, who was born on the 10th of March 1940.

Look at it over here:

That was it. Easy peasy, and very cute to do - even though there are so many very very serious use cases where pathfinding over a graph is actually a fantastic use case for Neo4j. People like TomTom, the Belgian Railroad, UGent, and many others use this capability for very serious use cases - and it is so so powerful.

If you want to learn a bit more about similar use cases on Wikipedia, in English, then please take a look at WikiDistrict, developed by Kernix using Neo4j.

Hope you found this as interesting as we did.

Cheers

Rik & Pieter