ReBeerGraph: importing the Belgian BeerGraph straight from a Wikipedia HTML page

I have written about beer a few times, also on this blog. I have figured out a variety of ways to import The Wikipedia Page with all the belgian beers into Neo4j over the years. Starting with a spreadsheet based approach, then importing it from a Google Sheet (with it's great automated .csv export facilities), and then building on that functionality to automatically import the Wikipedia page into a spreadsheet using some funky IMPORTHTML() functions in Google sheets.

But: all of the above have started to crumble recently. The Wikipedia page, is actually kind of a difficult thing to parse automatically, as it splits up the dataset into many different HTML tables (which makes me need to import multiple datasets, really), and then it also seems like Wikipedia has added an additional column to it's data (the "Timeframe" or "Period" in which a beer had been brewn), which had lots of missing, and therefore empty cells. All of that messes up the IMPORTHTML() Google sheet that I had been using to automatically import the page into a gsheet.

So: I had been on the lookout for a different way of doing the automated import. And recently, while I was working on another side project (of course), I actually bumped into it. That's what I want to cover and demonstrate here: importing the data, directly from the page, without intermediate steps, automatically, using the apoc.load.html functionality.

An easier, better, tastier (!) BeerGraphGuide in the Neo4j Browser!

I recently wrote two blogposts about a fancy new way to create "Guides" right in the Neo4j Browser - a system that we at Neo have used ourselves to create learning experiences for the Neo4j Graph Database - but that we are now also extending and making available to our community and Enterprise customers. The first post showed you what the result was, and the second post showed you how easy it was to create such a guide. And of course, since this is still all very new and under development by the inimitable Michael Hunger, it is still evolving at warp speed.

So that's why I am adding a third post to this series, based on the lastest version of Michael's github repo. In this version, he has added a much more convenient way to serve the HTML page that we use as the basis for the Browser Guide: we essentially don't need a CORS-enabled webserver anymore - we can just use the webserver that is already baked into Neo4j and which the Browser already uses. Michael did this by developing an unmanaged extension that picks up the HTML file from a local directory, and then serves it when you call the :play command in the browser. So how does it work? Let's show you.

Installing the Unmanaged Extension

You will find the extension on the github repo under the "guide-extension" directory. Michael has added some detailed installation instructions if you would be able and willing to compile the software yourself - but he has also added a compiled version link for Neo4j Enterprise 2.3.2. You can download the .jar file, and then just add this to the right directory of your Neo4j Enterprise 2.3.2 installation: the "plugins" subdirectory.

The BeerGraphGuide in the Neo4j Browser: the making of!

So last week I showed off this little tidbit that I created with the help of Michael and Oskar - a BeerGraphGuide right into the already fabulous Neo4j Browser.

So obviously I need to explain HOW I created that - and that's what I will do in this blogpost.

Creating a Browser Guide - easy!

Starting point of the exercise is Michael's github repo with all the source material. Download the zip file with all sources and unzip it on your local machine. The instructions online are kind of limited, but in principle the sequence of what we need to do is EASY:

1. we need to create an Asciidoc file (identical to what we use for our GraphGists) that describes the Neo4j Browser Guide.
2. we need to convert that Asciidoc file to an HTML file with a very simple shell script (see below).
3. we then need to figure out a way to LOAD that HTML file into the Neo4j browser. in a :play pane (similar to the built in guides that we already have)

So let's get cracking with that.