GET INTO the GRAPH

It's that time again. That Karaoke time. After having had the King of Pop, The Boss, and many other celebrities, it was time for the Queen of Pop. I chose this song, just to GET YOU INTO THE GRAPH. Seemed appropriate. Here goes:


Hope you enjoyed it.

Rik

The QCON Graph

Next week is QCON, London's holiest of holiest (according to some) developer conferences. It's going to be a ton of fun, and the lineup of speakers and sessions is just... impressive. Take a look at it over here - there's dozens of sessions that I would love to attend, by speakers that I would love to see. Unfortunately, I will probably miss most of it - as Neo4j is sponsoring the event, and that means BOOTH DUTIES! Yej!

But, not to be worried, there's fun to be had - with the schedule. I mean, who can read any of this stuff, really:

Tables, Schmables!!! Let's but it into a graph!

Start with a model!

From the tables above, I distilled the following model:

It's actually pretty rich: 

• Floors, days and times are connected to eachother by "in-graph-index" relationships. 
• Rooms and talks are there
• Talks are part of a track
• Persons can act as speakers and as track hosts.

There's some fun to be had there. 

Importing it into Neo4j

Naturally, I started with a spreadsheet. I needed to do some copy/pasting and cleaning of the data, and that's what I use that for. But once it was there, generating the Cypher to create the graph was trivial. Here's the create script to do that yourself - just clone it if you want.

Once we had that, we could start doing some querying on the graph. I have put some sample queries over here.

Here are some interesting ones exploring the graph:

This is a little overview of the "conference timeline: just exploring the days and then looking at the timeslots available in that day:

Here we go looking at some talks and tracks and how they are connected to eachother:

And this is probably my favourite: looking at the connectivity between the unfollowable Mark Needham and Yan Cui of Gamesys:

Of course there's plenty more to explore. That's why my friend and colleague Michael Hunger was kind enough to put the database (in read-only mode) on one of his servers - you can take a look at it over here.

Hope you found this useful - see you next week at the conference!

Cheers

Rik

Graph Karaoke - prepping for a big party!

Don't know if you know this, but in a few months from now, we are going to have the biggest Graph Party in Europe this year. GraphConnect is *ON* in London on May 7th, and you should all register. There will be talks, trainings, tracks by users, community contributors, briefings - and above all, LOTS OF FUN. Which is why I thought I would share a video that I recently saw with an entire TV studio just boasting lots of FUN:

This is, of course a WONDERFUL song, and there's "a capella" versions of it too, but of course I had to prepare a Graph Karaoke version of it too. Hoping to share that one with all the folks coming to London in May.

Hope you enjoyed that one. Play it LOUD!!!

Cheers

Rik

Innovation Pitch

Some companies are interesting. I mean, I myself have been working in startup environments for (djeez! has it been so long!) decades now, but some large organisations are equally interesting - especially in today's "information age". Every so often I get to meet fascinating people that are working in an industry that is literally being thrown upside down because of the modern technology swell of connectedness, mobile information, demanding customers, and innovative applications. After years, centuries sometimes, of successful business ventures in the "good old days", they find themselves in a place where they are sitting on wonderful assets, with real value, but also facing a growing need to re-assess how it all fits in this new age of digitalism. They need to innovate.

Innovation is a hard nut to crack. I am not an expert, but when I read the "Innovator's dilemma" a few year's ago, it became blatantly clear to me that innovation does not come natural to a large organisation. It simply doesn't. There's all kinds of internal and external forces that actually make it tremendously hard for large organisations to truly innovate.

That's probably why I personally find Startup organisations more my cup of tea, but it's also why I am truly impressed and greatly sympathetic when I see large organisations make a truly consolidated effort to innovate.

Yesterday, I was part of such an effort. Wolters Kluwer, global publishing powerhouse with a long standing history, headquartered in the Netherlands, organised an Innovation Pitch event for their executive team. Almost all of their board members and execs were there, and I had 10 minutes to "pitch Neo4j". Interesting.

I thought about this a bit - and I decided to go for the "high road". The pitch was not meant to sell product, not meant to position Neo4j even - but really was geared to getting these top-level international execs to think differently - to open up their minds to the wonderful world of graphs. I used the example of "How Wolves Change Rivers" to help illustrate that - as seen over here, or in the GraphGist over here.



I recorded the pitch at home - see below. The actual presentation included some Q&A and took a bit longer in total - but it was pretty much like this:



Slides are over here:



Probably a ton of other things that I could have said - but my main goal was to be remembered and get a conversation going with Wolters Kluwer. I would love to get your feedback, if any.

Cheers

Rik

Graph Karaoke using "Natural Language Analytics": Billie Jean

Last week, my friend and colleague Michael wrote a really interesting blogpost on natural language analytics using Neo4j. He used the One Ring poem as an example of how you could use Cypher to analyse a text file and put it into a Neo4j database for some advanced analytics. That immediately made me think about my Graph Karaoke Playlist, and how I could use this technique for some more Graph Karaoke generation. Wouldn't that be nice? More graph karaoke == good!

So in this post I will show you how easy it is to get this done. A couple of quick steps is all what is needed. Let's run through it and show you how it's done.

Loading a song

The first thing to do, as always, was picking a song. So this time, my kids picked it:
Billie Jean, by the King of Pop himself. Not wanting to sound pretentious, but I think it's great for my kids to big fans of that kind of music - seems like all of our educational efforts are yielding some results :) ...

Then I picked up the lyrics of the song over here, and put it into a google doc. The reason why, is that I wanted to do one small manipulation to the file in order to be able to use it for Karaoke: I added the Songpart and the Songpartsentence in two additional columns. Plus: the Google sheet has a very easy conversion into a csv file that we can then point the Load CSV process to.

Customizing the query

With that CSV file available, I then proceeded to customize Michael's query. Here it is:

 //create the karaoke graph  
 load csv with headers from "https://docs.google.com/a/neotechnology.com/spreadsheets/d/1DLu2bl1ZO7Zm8zU1UXNCDZGxsnBkicAJD4J-FSbVXLE/export?format=csv&id=1DLu2bl1ZO7Zm8zU1UXNCDZGxsnBkicAJD4J-FSbVXLE&gid=0" as csv  
 with csv.Songpart as songpart, csv.Songpartsentence as songpartsentence, csv.Songsentence as row  
 unwind row as text  
 with songpart, songpartsentence, reduce(t=tolower(text), delim in [",",".","!","?",'"',":",";","'","-"] | replace(t,delim,"")) as normalized  
 with songpart, songpartsentence, [w in split(normalized," ") | trim(w)] as words  
 unwind range(0,size(words)-2) as idx  
 MERGE (w1:Word {name:words[idx]})  
 MERGE (w2:Word {name:words[idx+1]})  
 MERGE (w1)-[r:NEXT {songpart:toInt(songpart), songpartsentence:toInt(songpartsentence)}]->(w2)  
  ON CREATE SET r.count = 1 ON MATCH SET r.count = r.count +1  

Let's run through this query to make it easier for you to digest. We start with the "load csv" statement. We point to the csv download link mentioned above, user the first row as headers and identify that with an identifier called "csv".

 load csv with headers from "https://docs.google.com/a/neotechnology.com/spreadsheets/d/1DLu2bl1ZO7Zm8zU1UXNCDZGxsnBkicAJD4J-FSbVXLE/export?format=csv&id=1DLu2bl1ZO7Zm8zU1UXNCDZGxsnBkicAJD4J-FSbVXLE&gid=0" as csv  

Then we pull the csv into three different sets that we can address separately with separate identifiers:

 with csv.Songpart as songpart, csv.Songpartsentence as songpartsentence, csv.Songsentence as row   

Then we use the Cypher "unwind" operator to create separate rows out of the "row" collection, and call these rows containing lyrics "text". 

 unwind row as text  

Afterwards, we are gong to be using "reduce" to remove punctuation marks and then split the text into individual lyrical words:

 with songpart, songpartsentence, reduce(t=tolower(text), delim in [",",".","!","?",'"',":",";","'","-"] | replace(t,delim,"")) as normalized  
 with songpart, songpartsentence, [w in split(normalized," ") | trim(w)] as words  

Lastly, we want to write these words into the graph. In order to do that, we are going to use "unwind" to generate an in-memory index, and then stepping through every sentence to generate the sequences. We do that with "Merge", first for the words, and then for the relationships. On every relationship, we will "karaoke-ize" the graph by assigning "songpart" and "songpartsentence" identifiers to every relationship.

 UNWIND range(0,size(words)-2) as idx  
 MERGE (w1:Word {name:words[idx]})  
 MERGE (w2:Word {name:words[idx+1]})  
 MERGE (w1)-[r:NEXT {songpart:toInt(songpart), songpartsentence:toInt(songpartsentence)}]->(w2)  
  ON CREATE SET r.count = 1 ON MATCH SET r.count = r.count +1  

That was easy!

So where is the KARAOKE???

Hah! That's what you came here for huh? Well, here's the result. 

I have put the queries on a gist so that you can take a look at it yourself. If you have any comments, then please let me know!

Cheers

Rik

Prototyping a Graph Database

Last night we did a great meetup in Amsterdam using a new, untested format. We had about 25-30 people attending, to try and prototype a graph database quickly and efficiently.


You can find the slides (borrowed from earlier work done by the awesome Neo4j team, Ian Robinson most particularly) over here or embedded above, but here's the way we approached this, which I think could be a general way of doing things:

First: a Model

We spent a long time on what are some of the best practices in Graph Database modelling. Here are some of the key takeaways, in my opinion:

• many of the modelling principles are similar to classical use story modelling as I first encountered it two decades ago in OO modelling. Create a story, extract the nouns and verbs to create a basic skeleton of the model.
• think about your use cases and your query patterns long and hard. They will drive the model - you want to model for queryability.
• Don't be afraid of normalisation. In relational modelling normalisation is expensive - in graph modelling it is not as much. You will tend to have "thinner" nodes and relationships in your graph model (holding fewer properties).
• Don't be afraid of redundancy: it is very common to have redundancy in the model - within the same database. Look at the different ways to model email exhanges or marriage relationships - you immediately see there are more than one way of doing things, and none of them is necessarily better than the other. It *depends* on your requirements, your queries.
• Iterate over the model - and understand that as your business requirements change, you may want/need to change the model. 

Here's what I ended up working on: a very simple model for a Configuration Management database, that would store Users in Departments of Companies, using a Service that relies on Hardware and/or Software:

I created that "drawing" with the Alistair Jones' "Arrows" tool. I have put the markup file over here.

Second: a prototype dataset

We then had a great time applying that model to generate an actual Neo4j database using Christoph Willemsen's GraphGen. It's amazing how quickly you can get a running Neo4j server populated with some sample data that is worth the while exploring. Here's my GraphGen model:

 /// CMDB GraphGen Example :  
 (customer:Company {name: company } *50)  
 (dept:Department {name: {randomElement:['IT','Sales','Marketing','Operations','Finance'] }} *100)-[:PART_OF *n..1]->(customer)  
 (u:User {name: fullName} *100)-[:PART_OF *1..1]->(dept)  
 (customer)-[:USES {percentage: {randomFloat:[2,0,1]}} *n..n]->(serv:Service {name: {randomElement:['Web','ERP','CRM','Mail','Calendar','Files']}} *10)  
 (serv)-[:USES *1..n]->(soft:Software {name: {randomElement:['OS','Database','Webserver']}} *3)  
 (serv)-[:USES *1..n]->(hard:Hardware {name: {randomElement:['Storage','Processor','Memory']}} *2)Here  

You can also find it on github. Pasting this into GraphGen gives you something like this:

Pushing the green "populate" button then fires of a bunch of Cypher queries that will create the database in your running Neo4j server in a matter of seconds:

This whole process works absolutely smooth as ever. It allows you to get a database up and running extremely quickly. It's really great.

Third: some traversals!

Last but not least, we then spent some time looking at the generated datasets and exploring some queries. To do that, I first had to add a few indexes for the queries to be efficient. Effectively adding the indexes on the labels' properties that I wanted to look up starting points with for my traversals. 

Next, I can do some queries that are quite typical of the "Configuration Management" domain, and we often see in various kinds of "Impact Analysis" use cases. Basically it means traversing the graph from two ends:

• Starting from the Hardware/Software end: asking the question about who would be impacted if a particular piece of hardware/software would fail. An example query would look something like this:

 match (h:Hardware {name:"Storage"}),(u:User), p=AllShortestPaths((h)-[*..5]-(u))  
 where id(h)=865   
 return p;  

The result looks like this (the "Storage" device affected is at the center

• Starting from the User end: asking the question about what hardware/software pieces a particular user accesses. The query looks like this:

 match p=Allshortestpaths((u:User {name:"Leland Blanda"})-[*..5]-(h:Hardware))  
 return p  

The result looks like this - the User in case is at the very right, and the systems he/she uses to the left.

All these queries are on github as well. You may need to change the identifiers for users/systems as they will obviously be different if you generate your own database.

All in all I thought this exercise was a very useful and a pleasant one. Going from zero to graph in a matter of hours is pretty interesting, and offers a lot of potential for iterative experimentation/prototyping.

Hope you found this useful. As always, feedback very very welcome.

Cheers

Rik

Always look at the bright side of life!

A life lesson AND a great way to have fun together with one of the nicest colleagues: Nigel. He got me going on the Graph Karaoke thing, and now gave me another idea for a song that had to be sung, so he deserves the credit, and a picture:

As you can see, Nigel is a big fan of Karaoke, and his mission in life is to make you smile. He can't stop. It's amazing. Such a hero. And yesterday he was talking to me about how great it would be to have a Monty Python song in the Graph Karaoke playlist - so here we are:

Hihi. I guess I don't have a social life - and most definitely not when locked into a hotel room far away from home. 

Hope you enjoy it.

Rik