Same .CSV files, different neo4j database!

As you probably know, Neo Technology is getting closer and closer to releasing the major new release of the neo4j database: version 2.0. Going from 1.9.x to 2.0 is a bigger jump than you may expect. It's not just a 0.1 difference ;-) ... 2.0M06 is jam-packed with new features, among which the fantastic new extension to the property graph data model: node labels.

Labels are simply fantastic in my opinion. You can read up on the advantages that they bring, now and into the future, over here. But what does it mean to some of my previously generated neo4j databases? Well - turns out it's quite a thing. Not that the actual upgrade of the datastore is difficult (it's as simple as uncommenting the "allow_store_upgrade=true" line in the neo4j.properties file), but how could I actually start taking advantage of the labels feature, in my datamodel, in my queries, in the fantastic new neo4j browser? Let's find out.

Revisiting my Last.fm model

In my previous blog posts, I had imported a last.fm scrobbling dataset using different methods. The model looked something like this:

In all of the nodes of that model, I had included at minimum two properties: a "name" and a "type". So guess what: it makes total sense to convert these "type" properties into labels, ending up with 6 different subgraphs based on the labels: listeners, scrobbles, tracks, artists, albums and dates. Nice. I will be able to use the new, improved indexing that neo4j 2.0 features based on these labels, of course.

Re-importing the data from the same .csv files

So then I need to recreate the database reflecting this change. My source files (see the previous blogpost: just download from here) are of course the same - all I need was a slightly modified import process. My dear friend Michael Hunger has already prepared a 2.0 version of the neo4j-shell-tools - and they just work like a charm. 

Here are the import statements for the nodes:

import-cypher -d ; -i ./IMPORT/INPUT/nodespart1.csv -o ./IMPORT/OUTPUT/1out.csv create (n:#{type} {name:{name}}) return n.name as name

import-cypher -d ; -i ./IMPORT/INPUT/nodespart2.csv -o ./IMPORT/OUTPUT/2out.csv create (n:#{type} {title:{title}, name:{name}}) return n.name as name

As you can suspect, the n:#{type} piece is the interesting part. This is where we use the "type" data-element from the csv files for the labels, not for the old "type-property". The reason why the # is there is because Michael had to do some wizardry to allow for parametrized labels - which is normally not supported in Cypher.

Adding indexes based on Labels 

Before we now go and import the relationships, we have to add the indexes on these newly created nodes and labels. We do that as follows in the neo4j-shell:

CREATE index on :date(name);  

CREATE index on :album(name);  

CREATE index on :scrobble(name);  

CREATE index on :listener(name);  

CREATE index on :artist(name);  

CREATE index on :track(name); 

This takes a second or two. but once complete, we can see that the indexes are ready to be used by typing the schema command:

And then we can proceed to import the relationships, again with the same .csv files.

No longer starting with a start

Importing the relationships is also done with neo4j-shell-tools, but slightly different from last time: the parametrized cypher queries no longer "start with a START", they now start with a "MATCH". This is because, now that indexing has become an integral part of neo4j, you can really work with Cypher in an even more declarative fashion than before. You don't have to imperatively tell the database where to start - it will figure it out for you based on the pattern that you specify.

Here are the new import statements:

import-cypher -d ; -i ./IMPORT/INPUT/APPEARS_ON.csv -o ./IMPORT/OUTPUT/3out.csv MATCH (track:track), (album:album) where track.name={mbid1} and album.name={mbid2} create unique track-[:APPEARS_ON]->album return track.name, album.name

import-cypher -d ; -i ./IMPORT/INPUT/CREATES.csv -o ./IMPORT/OUTPUT/4out.csv MATCH (album:album), (artist:artist) where artist.name={mbid1} and album.name={mbid2} create unique artist-[:CREATES]->album return album.name, artist.name

import-cypher -d ; -i ./IMPORT/INPUT/FEATURES.csv -o ./IMPORT/OUTPUT/5out.csv MATCH (scrobble:scrobble), (track:track) where scrobble.name={scrobble} and track.name={mbid} create unique scrobble-[:FEATURES]->track return scrobble.name, track.name

import-cypher -d ; -i ./IMPORT/INPUT/LOGS.csv -o ./IMPORT/OUTPUT/6out.csv MATCH (listener:listener), (scrobble:scrobble) where listener.name={user} and scrobble.name={song} create listener-[:LOGS]->scrobble return listener.name, scrobble.name

import-cypher -d ; -i ./IMPORT/INPUT/ON_DATE.csv -o ./IMPORT/OUTPUT/7out.csv MATCH (date:date), (scrobble:scrobble) where scrobble.name={song} and date.name={date} create scrobble-[:ON_DATE]->date return scrobble.name, date.name

import-cypher -d ; -i ./IMPORT/INPUT/PERFORMS.csv -o ./IMPORT/OUTPUT/8out.csv MATCH (artist:artist), (track:track) where artist.name={mbid1} and track.name={mbid2} create unique artist-[:PERFORMS]->track return artist.name, track.name

import-cypher -d ; -i ./IMPORT/INPUT/PRECEDES.csv -o ./IMPORT/OUTPUT/9out.csv MATCH (date1:date), (date2:date) where date1.name={date1} and date2.name={date2} create date1-[:PRECEDES]->date2 return date1.name, date2.name

You can download the entire set of statements from over here.

So there we have it: a newly imported, nicely labeled neo4j-2.0 dataset. So let's fire up the browser and see what the result looks like?

Very nice! Now I can start playing around to my hearts' content with the new browser and have even more fun. How is that even possible?

Hope this was useful. Until next time.

Importing my Last.fm dataset - the neo4j way

Some time ago, I blogged about how you could create an interesting graph dataset in neo4j using the data from Last.fm. At the time, I used Talend as an ETL tool, to do the import into neo4j – as the dataset was quite large and the spreadsheet method would probably not cut it anymore. It worked great – the only downside (for this particular use case) was that ... I had to learn Talend. And not that that is terribly difficult – especially not if you are an experienced ETL professional, which I am clearly NOT – but there was definitely a learning curve involved. So: there continued to be a latent desire to do this import into neo4j natively – without separate tooling. And now, I think we have that, thanks to the ever-amazing Michael Hunger.

Enter neo4j-shell-tools

Michael created a collection of utilities that basically plug into the neo4j-shell, and extend its functionalities with things like... data import functionalities. There are different options, and you should definitely read up on the different capabilities, but for my specific Last.fm use case, what was important was that it can easily import the csv files that I had created at the time for the import using talend.

You can read up on the details of the shell-tools in the readme (in contains very simple installation instructions that you would need to go through beforehand – essentially installing the .jar file in neo4j's lib directory). Once you have done that and you shutdown/restart the neo4j server, you are good to go.

Creating the database from scratch.

As you will see below, the steps are quite simple:

Step 1: start with an empty neo4j database

What's important here is that the neo4j-shell-tools work on a **running** neo4j database. You do not need to introduce downtime, and you do not use the so-called “batchimporter” method – instead you are doing a full blow, transactional, live update on the graph, using this toolset.

Step 2: prepare the .csv files

I had already prepared these files for the previous blogpost – so that was easy. The only difference that I had to make was that I

• had to make sure that the delimiter that I was using was right. The neo4j-shell-tool allows you to specify the type of delimiter, and getting that wrong will obviously lead to faulty imports
• had to add a “header” row at the top of the text files. The neo4j-shell-tool will assume that the first line of the .csv files defines the structure of the rest of the file. Which also then means, that I needed multiple files as both the nodes and relationships that I wanted to add have a different structure/type.

So I ended up with 2 .csv files to add nodes to the graph, and 7 .csv files to add the relationships between the nodes. You can download everything here.

Step 3: prepare the import commands

The node import commands look like this

import-cypher -d ; -i nodespart1.csv -o 1out.csv create (n{name:{name}, type:{type}}) return n.name as name

import-cypher -d ; -i nodespart2.csv -o 2out.csv create (n{title:{title}, name:{name}, type:{type}}) return n.name as name

The structure of these commands is fairly simple:

• import-cypher: calls the shell tool that we want to use
• -d defines the delimiter of the file that we are importing. In these case a “;”.
• -i defines the input file. On OSX, not adding a path will just look for the file in the root of your neo4j installation directory. In many cases you will want to have an absolute, or relative path from there.
• -o defines an option output file where the result of the import commands will be written. This is intended for logging purposes.
• And then finally, with the highlighted “create...” section, we basically define the Cypher query that will do the import transaction – using the parameters from the csv file (between { }) as input.

Note that the neo4j-shell-tools provide some separate functionalities for dealing with large input files and for tuning the transaction throttling (how many updates in one transaction), but that for this purpose we really did not need to do that.

Then for the relationship import commands, we have a very similar structure:

import-cypher -d ; -i APPEARS_ON.csv -o 3out.csv start n1=node:node_auto_index(name={mbid1}), n2=node:node_auto_index(name={mbid2}) create unique n1-[:APPEARS_ON]->n2 return n1.name, n2.name

import-cypher -d ; -i CREATES.csv -o 4out.csv start n1=node:node_auto_index(name={mbid1}), n2=node:node_auto_index(name={mbid2}) create unique n1-[:CREATES]->n2 return n1.name, n2.name

import-cypher -d ; -i FEATURES.csv -o 5out.csv start n1=node:node_auto_index(name={scrobble}), n2=node:node_auto_index(name={mbid}) create unique n1-[:FEATURES]->n2 return n1.name, n2.name

import-cypher -d ; -i LOGS.csv -o 6out.csv start n1=node:node_auto_index(name={user}), n2=node:node_auto_index(name={song}) create n1-[:LOGS]->n2 return n1.name, n2.name

import-cypher -d ; -i ON_DATE.csv -o 7out.csv start n1=node:node_auto_index(name={scrobble}), n2=node:node_auto_index(name={date}) create n1-[:ON_DATE]->n2 return n1.name, n2.name

import-cypher -d ; -i PERFORMS.csv -o 8out.csv start n1=node:node_auto_index(name={mbid1}), n2=node:node_auto_index(name={mbid2}) create unique n1-[:PERFORMS]->n2 return n1.name, n2.name

import-cypher -d ; -i PRECEDES.csv -o 9out.csv start n1=node:node_auto_index(name={date1}), n2=node:node_auto_index(name={date2}) create n1-[:PRECEDES]->n2 return n1.name, n2.name

Note that, because of the domain model that we have from the last.fm dataset, some relationships have to be unique and others don't – hence the difference in the Cypher queries.

Step 4: executing the commands

Then all we need to do is to put the files on the right locations, make sure that autoindexing is correctly defined, and then copy/paste the commands into the neo4j-shell.

On my MacBook Pro, the entire import took about 35 seconds, and I ended up with the database that I had previously created with the Talend toolset:

And then the same graphic/query exploration can begin. You can take the graphical tools for a spin, or alternatively create your own cypher queries and get going.

Conclusion

Overall, I found this new process to be extremely intuitive and straightforward – even simpler then what I had experienced using the Talend toolset. I have put the zip-file and the corresponding input statements over here – so feel to download and experiment yourself. Just make sure that you put the .csv files in the neo4j “home directory”, or adjust the paths as you want (both relative and absolute paths seemed to work on my machine).

Hope this was useful. Until the next time!

Rik

Podcast Interview with Nicole White, Neo Technology

Here's another fantastic episode of our Neo4j Graph Database Podcast: I had a super nice late-night (for me) conversation with Nicole White, a colleague of mine in our San Mateo office. She is a Data Scientist at Neo, which means she helps us out with a lot of our internal data questions - and develops some fantastic tools for that. She also frequently speaks at conferences and meetups, and writes stuff over here, here (super cool Flask tutorial btw!) and here.

Here's the episode for you:

Here's the transcription of our conversation

RVB: Hello everyone. My name is Rik - Rik Van Bruggen - from Neo Technology, and here I am again recording another episode of our Neo4j Graph Database podcast. With me tonight is - all the way from California - Nicole White, from Neo. Hi, Nicole. 

NW: Hi Rik, how are you? 

RVB: I'm very well. And yourself? 

NW: Very good. 

RVB: Very good. Well, it's late at night for me, it's still afternoon for you, but I thought I'd take the opportunity to talk to you a little bit because-- well, maybe you can explain that, yourself? Who are you, and what do you do with Neo? Do you mind explaining that to our listeners? 

NW: Right. Yeah. My name is Nicole White, I'm a data scientist at Neo4j, and we actually use Neo4j internally to hold all of our data that we collect - marketing, sales, product usage. Particularly with Neo4j, I'm using Neo4j to  perform common data science tasks, but Neo4j is our data storage solution. All of our data sits in one spot, one nice clean spot, and thus it's very easy to answer some of the complicated questions that we weren't able to answer before. Actually, all of our tools that I've built out internally are built on top of Neo4j, which is probably my favorite part about my job - is that I get to use Neo4j, I don't have to touch SQL ever, I just get to write cypher all day long which is super, super fun. So with regards to Neo4j, that's who I am. But I just recently graduated from grad school with a degree in statistics, and before that I got an undergraduate degree in economics and math. Just hailed from Austin, Texas, moved here to California, San Mateo, ten months ago, I think, is when I started. I'm coming up on my first year here at Neo. 

RVB: Okay. Well, this sounds like we're eating our own dog food, right? Using Neo for a-- 

NW: Yes, we are. We actually just upgraded to 2.2. All of our systems were just upgraded to 2.2. 

RVB: Fantastic. How did you get into Neo, Nicole? I mean, you must have started using that at grad school or at university, or how did you get into it? 

NW: Yeah. It was actually the GraphGist Challenge. It was the very first one. I saw it on Twitter. Someone who I was following re-tweeted a Neo4j tweet about the GraphGist challenge, and so I looked at the page and I saw a GraphGist. I think the first GraphGist I saw was something about doctors and prescriptions or something, and I saw Cypher and I was like, "This looks really cool." And, of course, there is an opportunity to make money so I was all about it. I looked at Cypher-- 

RVB: Typical student, right [chuckles]? 

NW: I know, right [chuckles]. I was actually, at the time that I came across these GraphGists, I was working on a project with My Flights data set in school with all of the-- it was the Bureau of Transportation's statistics, all their data on delayed flights across all domestic-- US domestic airports. I had that all in Oracle database - a SQL database - and I was doing just like some pretty basic analysis on it for a school project, and then as soon as I saw Neo4j-- as soon as I saw Cypher, I already knew that a lot of my SQL queries would be so much easier in Cypher. I was already seeing that I would prefer to have Neo4j.  So I moved it all to Neo4j, and then I also created that GraphGist of the flights, and that's the first data set that I learned Neo4j on and learned Cypher on.
RVB: Yeah, fantastic. So you mentioned that you thought , you know-- 

NW: The GraphGist on. 

RVB: Yeah. You mentioned that you thought that it would be a lot simpler than in SQL, that in SQL. Did that turn out to be true? Is that-- 

NW: Yeah. 

RVB: --one of the things that you like about it, or where does the love for Neo come from? 

NW: That has to be the first thing that hit me, was there are some SQL queries that I really struggled with. There was one - it was so simple to say in English. It was just like,  "I want to see airports that are, by definition, span multiple states." Because some airports are technically-- some over in the DC area, they technically sit in several states somehow, and writing that query in SQL was strangely hard. I had to use a partition by and something weird. I remember it was that query specifically, and when I saw Cypher, I was like, "That's going to be super easy," and it was. I took a SQL query that was probably, like, 20 lines and really hard to read, and put it into Cypher. And that's what I love about Neo4j, is that you can take a question that you've posed in English and very easily translate it into Cypher and vice versa. Like, I can take a Cypher query and then translate it back to English very easily, even if it's a data set I've never seen before, a Cypher query I've never seen before. I can easily scan through it and say, "This is what they're doing," in English, whereas when someone sends you a SQL query and that-- particularly with a data set you haven't worked with, translating it back to English is really hard. Just trying to scan everything that's going on in the cycle-- or in the SQL query, I think is very difficult. So I think from a collaboration standpoint, Neo4j is super awesome. Because I got a few of my classmates to work with me on this so I'm putting all the flights data into Neo4j, and just collaborating across queries was much simpler because we could understand what-- 

RVB: Because of the readability, yeah. 

NW: The readability is just a huge factor for me, and I think that's probably what I love most about Neo4j, is Cypher, I would have to say [chuckles]. 

RVB: Yeah, very cool. You mentioned earlier that you were using it for data science and I believe you're also doing a lot of talks on integrating Neo with R right, with the R project. Can you tell us a little bit more about that maybe? 

NW: Yeah, so I wrote the R driver for Neo4j. It's called RNeo4j, and I use that internally here as well a lot as in addition to Python. Python and Neo4j do all the heavy work, and then any reporting, or analysis, or charting visualization stuff, I'll spin up my R driver and pull Neo4j data into R for more fancier statistic stuff which we've been doing recently for some new projects that we've just started here at work. The R driver, essentially, is just a wrapper for the rest API and it will pull Cypher query results into your R environment very easily, and then you can-- and that opens up a lot of doors for analysis purposes. But yeah, I've been doing a lot of talks around that. I do a meet-up on the R driver probably like once every couple of months here in the Bay Area-- 

RVB: You should do that in Europe, Nicole. I mean-- 

NW: I should [laughter]. I'll be in Europe soon. I'll be there for a GraphConnect London, so I'll probably do something with Mark while I'm over there, because he uses the R driver probably more than I do. If you look at his blog [chuckles]-- 

RVB: Yeah, exactly. So maybe wrapping up, Nicole, where do you think this is going? Where do you hope, where do you want it to go, and where do you think it will go? The evolution of graph databases is so quick this days yet-- but what do you think [chuckles] is coming at us right now? 

NW: I just think we get to look forward to just a huge improvement in user experience from a user standpoint. I've been a user of Neo4j for a little bit over a year now and it's just crazy how quickly they improved the user experience. Just from 1.9, I think, is when I first saw it, 2.0 was huge, just the Neo4j browser has gotten so much better with the 2.2 release. 

RVB: Absolutely, yeah. 

NW: There's just so many nice, convenient-- like they're subtle, the changes are subtle, but when you're a super-heavy user of the Neo4j, they really stand out. They've made some subtle changes to the Neo4j browser that I really like. I think, I'm mostly looking forward to the huge improvements in user experience that are most likely continuing to come. Also, I think from my standpoint as well, the whole import process for Neo4j is going to continue getting more awesome because I feel like import was our biggest weakness when I first encountered Neo4j. We didn't have a lot of really easy-to-use tools. And within a year, now we have load CSV, which is really easy, and then we have the 2.2 import tool, which is really easy and super-fast. I feel like that's also what I'm looking forward to, is continued improvements on the import part of Neo4j, because that's the first part you're going to encounter, right? As a new user, the first thing you're going to do is import your data, so I'm really happy that we've been putting so much work into that part. The whole import experience has gotten much better. 

RVB: I could not agree more [chuckles]. As you know, I'm in sales, and I know how important this is. Very good, thank you so much Nicole for coming online and doing this recording with me, I really appreciate it. 

NW: Thanks for having me. 

RVB: It was great having you on the podcast and I really appreciate it. Thank you again, and yeah, I look forward to seeing you at GraphConnect. 

NW: Yeah, I look forward to it as well. Have a good one. 

RVB: See you, bye.

Subscribing to the podcast is easy: just add the rss feed or add us in iTunes! Hope you'll enjoy it!

All the best

Rik

Podcast interview with Stefan Plantikow, Neo Technology

Today's episode in the Graphistania podcast is one that I have really been looking forward to, for many reasons. First of all, our guest is such a lovely guy - feels like I could go out on a VERY long pub crawl with Stefan - seriously. Then, he has been working on some of the most interesting topics in Neo4j - another bonus. Most recently, he has worked on the "swiss army knife" of Neo4j tooling, the Awesome Apocs. Enough reason to have a good podcast chat together - and here that is:


Here's the transcript of our conversation from July 4th, 2016:

RVB: 00:02.518 Hello everyone, my name is Rik, Rik Van Bruggen from Neo, and here we are again, recording another Graphistania podcast, and today I have one of my lovely colleagues from the engineering team with me, Stefan Plantikow from Berlin. Hi Stefan.

Podcast Interview with Wes Freeman, Information Analysis, Inc.

It's been a while since the last podcast, what with GraphConnect and all happening last week in London.

It was a supremely busy and FUN week, but now of course we need to get back to "business" as usual - which means publishing another great chat with one of our long standing active Community members, Wes Freeman, of Information Analysis, Inc.

I first got to know Wes from his blog and his excellent work on Cypher (note: this site is a bit outdated now, as it dates from back before Cypher had many of the performance optimization features (eg. the cost-based planner) - but it's still a good read :)). He's a generally-all-round-good-guy - and we had a great chat:

Here's the transcript of our conversation:

RVB: Good morning everyone. This is Rik, Rik Van Bruggen from Neo Technology, and here we are again recording another session for our Neo4j graph database podcast. And it's a remote session again, all the way across the Atlantic. On the other side is Wes Freeman of Information Analysis. Hi Wes. 

WF: Hi. 

RVB: Hey, good to have you on the podcast. 

WF: Thanks. 

RVB: That's great. For those listeners that don't know you Wes, would you mind introducing yourself quickly? 

WF: Sure. I'm a technologist. I guess recently CTO of emerging technology at Information Analysis, and I've been looking at Neo stuff for almost three years now. It's kind of amazing. 

RVB: Time flies when you're having fun, right? 

WF: [chuckles] Indeed. I've done a fair bit of stuff with Neo, but maybe we'll get into that momentarily here. 

RVB: Absolutely. You've been a very active member in the community, right? I remember you posting a lot of stuff on how to do cypher queries, and stuff like that. That seems to be an area of expertise for you? 

WF: Yeah. I think I kind of like problem solving in general, and cypher kind of clicked with me, so I really enjoyed helping people try to get their cypher to work. That was like a fun pastime for more than a year [chuckles]. 

RVB: Have you done a lot of work with the recent new cypher planner in 2.2? 

WF: Yeah. I can't say I've done a lot of work with it, but I have gone through and it looks excellent, the way they've changed the explain and profile stuff. I really love the way it draws it out in the browser for you, especially. 

RVB: Yeah, that's cool. 

WF: But yeah, the cost-based optimizer is shaping up, and I really think things are awesome these days, so we're getting up there in cypher speed. 

RVB: Yeah, that's fantastic. Wes, how did you get into Neo? How did you get into graphs in general? Do you mind giving us a little bit of the history there? 

WF: Sure. It all started with a personal project, which is still ongoing. Basically, I was trying to analyze and keep records of decision trees for the game of chess, and I started various databases. I eventually stumbled on MongoDB and I tried-- you can keep a small hierarchy in MongoDB, because you can nest the documents. Then what I ended up running into problems with, was I needed to find out whether I'd seen a position in the decision tree before. When you get 40 or 50 moves deep, I was basically having to query up the tree in my own-- get the parent, get the parent, get the parent. It just wasn't performant. And it just so happened that I was on Meetup, and Andreas Kollegger had scheduled the second Meetup of Baltimore, D.C. graph databases. It was in a library, and I have to say it was a fun time, but there was no projector, so we were all sharing slides around. It was kind of funny-- 

RVB: Pretty old school, huh [chuckles]? 

WF: Oh yeah. But it didn't matter, the content was very exciting and everybody dug into it. Ended up talking to Andreas. I was like, "You've got to have more of these Meetups, maybe next month or something." He's like, "Oh." Then later I get this email, "So how'd you like to be the co-organizer of the D.C. Meetup?" 

RVB: Be careful what you ask for [chuckles]. 

WF: Yeah. 

RVB: That's pretty cool-- 

WF: It was fine. That's how I got into that. 

RVB: So what do you like about it? What attracts you to graphs and graph databases? We've had a lot of people explain that on this podcast, so what's your perspective? 

WF: For me, the selling point initially was the performance of deep traversals. That's where I got into it. But as I got more into it, you can do-- it's got this also other selling point of-- it's very flexible, and you can model any domain in it. That's another aspect that I like a lot. And I like cypher. Cypher makes things easy to query. I also get into the job at API and write some unmanaged extensions. 

RVB: You've been around [chuckles]. 

WF: I’ve done all the stuff, yeah [chuckles]. I haven't just done cypher, but it's definitely nice to use when you can. 

RVB: Where do you think-- where are you guys, and where are you guys going to take graphs? Also where do you think graphs are going to go in the industry? Any perspectives on that? Where do you think we'll be in a couple years from now? 

WF: It seems like it's growing like crazy, so that's great. I've seen lot of people-- I'm still running the Meetup, and we've seen pretty good growth for the last three years, or two and three quarter years. And we see new people all the time, so I think that's also indicative that it's spreading. At least in our local area I can confidently say that it's growing, and I see buzz on the Internet. I think definitely seems promising. I'm investing my time in it. 

RVB: We thank you for it [chuckles]. I think it's starting to become more and more prevalent in the industry as well, right? With customers and clients? 

WF: Yeah. 

RVB: Very good. Well, Wes, thank you so much for coming on the podcast. I'm going to wrap up here and keep this podcast nice and short. But it's been a pleasure talking to you. Thank you. 

WF: Sure thing. Thanks a lot. 

RVB: I look forward to seeing you at one of the Meetups or conferences in the upcoming months. 

WF: Yeah, I'm sure I will. 

RVB: Thanks, Wes. 

WF: All right, bye.

Subscribing to the podcast is easy: just add the rss feed or add us in iTunes! Hope you'll enjoy it!

All the best

Rik

Podcast interview with Johan Teleman, Neo4j

I had a great time chatting to my colleague Johan Teleman, recently. Johan works at the Neo4j Engineering team in Malmö, and has been doing some great work - on Cypher performance among other things. As it turns out, there's a LOT that has been done already (look for some spectacular stuff in Neo4j 3.4), but there are so many interesting plans for the future as well. Here's our chat:

Here's the transcript of our conversation:

RVB: 00:01:39.301 All right. Hello, everyone. My name is Rik, Rik Van Bruggen from Neo4j, and here I am again recording another episode for our Graphistania podcast. And today I am very happy to have one of my Malmö colleagues on the other side of this call. That's Johan Teleman. Hi, Johan. 

JT: 00:01:59.902 Hi, Rik. Happy to be here.

Podcast Interview with Andrés Taylor, Neo Technology

If you have been following our podcast, then you probably know by now that there are some exceptional people in and around the Neo4j ecosystem. It's pretty amazing - it feels like a great honour and privilege to be part of it. Today's podcast episode is going to be another session with an exceptional character, someone that you really don't want to mess about with - for good reasons:


O yeah. Andrés Taylor, one of the lead engineers at Neo Technology, is an avid Jujutsu practitioner - and all round amazing guy and splendid engineer. He is also known as the "father of Cypher", the declarative graph query language of Neo4j. So let's talk about that a little:


Here's the transcription of our conversation:

RVB: Hello, everyone. My name is Rik van Bruggen from Neo Technology. Here I am again, doing another recording for our podcast on Neo4j and graph databases. It's another remote session over Skype with someone that most of you probably don't know yet, but you should. That's Andrès Taylor from our dev team. Hi, Andrès. 

AT: Hi, Rik. 

RVB: Hey. Good to have you on the podcast. For those of you that don't know this yet, Andrès is one of the leading developers on the Neo4j development team. I can probably call you the inventor of Cypher. Right, Andrès? 

AT: Okay [chuckles]. You can say that. 

RVB: So, would you mind introducing yourself a little bit, Andrès? 

AT: Sure. So, like you said, I'm working in the dev team. I'm working on Cypher, the execution engine. The thing that takes Cypher query and actually runs it. I'm also the head of the Cypher language group which is working with the language part, the user facing side. The semantics of the language more than the implementation of the language. 

RVB: All right. How long have you been with Neo on this? 

AT: This is my-- four and a half years. 

RVB: Wow, you're a veteran [chuckles]. 

RVB: Excellent. And, you're based in Malmö, Sweden. Could you tell me a little bit about what attracted you to graph databases, and what do you love about it? What do you love about Cypher, as well? That's also a really cool thing for our listeners, I think. 

AT: I'll give it a try. Before I joined Neo, I had two things that I had done a lot, which were either agile consulting or databases. I spent a lot of time working as a DBA, performance tuning people's databases. 

RVB: You mean a SQL DBA then? 

AT: SQL database administrator, especially on Microsoft SQL server. And so, I would go in and help people with their queries and make them fast. When I started working with Neo4j, I was blown away by the data structures, the access pass that you could take through your data. It opened up ways of looking and working with the data that a SQL database just couldn't give you. 

RVB: This was a an early version of Neo at that time. Yeah? 

AT: I think I joined-- so the first commit that was included in a Neo4j release was 1.2, I think. 

RVB: So the access path, you mean you know the power of the queries, right? Is that what I'm sort of hearing? 

AT: Well, not really. [laughter] That was the problem that-- when I started looking at Neo4j and working with Neo4j was very, very quick. It was super easy to write really performant queries. But the queries needed a lot of hand-holding. You had to do a lot of the work that for someone coming from a SQL background, the query planner does for you. 

RVB: That was an imperative approach to queries. Is that right? 

AT: Exactly. The traversal framework that was the main use of querying databases before Cypher is something that is very imperative in nature. You describe where to start. You describe which path to go through and where you want to end up, and when to do filtering. Stuff like that, you have to make an explicit decision around. So, that's where I started. I thought it was awesome in performance power, but it was kind of difficult to work with. Especially when you came back to the code - the traversal code - after you've written it. You kind of hold it in your head while you were writing it, but then coming back to it was really difficult to understand what you were thinking at the time. 

RVB: So, I feel the birth of a declarative query language coming up here. 

AT: Yes [chuckles]. 

RVB: [chuckles] How did that come about? Tell us about that. 

AT: Cypher was this third attempt I did at a query language. First I started by doing a DSL in Java to try to express your queries, and little bit higher level than what the traversal frame work gave you but that was super difficult then, not pretty at all. Then I did JavaScript wrapper around the API so you could get a REPL. You could go in and try your queries live without having to bait the little program. And I added a little bit of sugar around the graph database API but still, that was not very useful. And then, we started sending a text file around with examples of how do you wish you could express your queries. Me and Mattias were working with-- I always had a couple of people in the office. No one really took it seriously, because none of the clients were using it or were interested in it. And it was difficult to get any interest from higher-ups in the organization. It was not something we were selling at the time. 

RVB: I seem to remember that there was something with Scala, as well. 

AT: Right. Because it wasn't a super important project from the organization, it was something that we spent-- I spent the 20% time that we got, and weekends and evenings, working on this stuff. If I'm working evenings, I'm not working in Java. So, I looked around. I was looking for something better than Java. I've worked in Clojure, but Clojure was very remote-- far away from Java. So, Scala it was. 

RVB: So you wrote and still write the Cypher part of Neo4j in Scala? Right? 

AT :Yes. All the compiling of a query is done in Scala. 

RVB: Super. Pretty cool. So that's very interesting. Can you tell us a little bit more about the future, Andrès? Where is your part of Neo4j and where is Cypher going? Would you mind sharing a little bit of light on that? 

AT: Sure. The language changed a lot in the first few versions. Since the 2.0 release, it's stabilized quite a lot. We have not added a lot of constructs. The language, we haven't added many new functions or features to it. And I said, we've been focusing on making what we have run as quickly as possible. It doesn't matter how pretty a language you have. If it runs slowly, no one's going to use it. 

RVB: Exactly, yeah. 

AT: So that's what we spent big parts - most of 2014 - working on. That should be visible in the 2.2 release coming up now. And, in the immediate future we have more of that. There's more performance stuff that we want to do. We want to look at-- hopefully, we want to get to generating code for execution plans and compiling it. That should give a nice performance boost. 

RVB: Is that a little bit like a stored procedure time thing, or am I reading that wrong?
AT: No. The product of the compiling is something that you can run. And when you run code you can either run it in an interpreted mode or a compiled mode. You've heard those terms before? 

RVB: Yes, I have. 

AT: What we have today is an interpreted version of Cypher. We build the tree structure and we execute that tree structure. And we need to interpret it every time we come across it. What we want to do instead is to actually generate Java code, which we dynamically compile and load, and execute. 

RVB: Super interesting. That should give us a big boost in performance. Any other big things that are coming in the future, do you think? 

AT: That's from the implementation perspective, that's something that we've spent quite a bit of time on. We're interested in how to distribute this, and to running more - either servers or threads - on it. For long running queries, not for the short-lived ones that just take a couple of milliseconds. There's not much point in distributing it. 

RVB: No. But things like PageRank and betweenness calculations, those types of things, you're talking about, right? 

AT: Exactly. Analytical queries. So that's something that we talked about, thought of. And then, on the other side of things is we want to add more indexes. We have-- we want to add text searching. We want to do stuff around dates. There's a lot of features that-- I mean we've spent a lot of time making the engine run smoothly. And now, I think it's time to start adding up new bells and whistles to the language as well. 

RVB: Andrès, there's so many things we could talk about. I do want to keep these podcasts a little short and snappy. So, any other final remarks you want to give our listeners? Or should we keep it at this? What do you think? 

AT: No, that's-- I don't have anything else to add [chuckles]. 

RVB: Very cool. Well, in any case, I really thank you so much for coming online and doing this little recording with me. This is super nice. 

AT: Thank you for having me, Rik. 

RVB: Yeah. It's fantastic. I'll look forward to all the wonderful things that you guys are working on. It's made a big boost in 2.2. And I'm sure it's going to be even better in the future. Thank you. Thanks a lot. 

AT: Thank you, Rik. 

RVB: Okay. Have a nice day. Bye. 

AT: Bye-bye.

Subscribing to the podcast is easy: just add the rss feed or add us in iTunes! Hope you'll enjoy it!

All the best

Rik

Part 2/2: Graphs are Bloom-ing

Earlier I wrote about how I connected the newly announced preview version of Neo4j Bloom to my good old faithful Belgian BeerGraph. See part 1 of this 2-part series for that story. I actually split up the story into two parts, because I feel like there's a super interesting and powerful part to Bloom that deserves a bit more attention: the mechanism of the custom Search Phrases.

As we mentioned in the previous post, Bloom structures your exploration and discovery into specific "views" on the graph data, called "Perspectives. You can select the perspective you find most appropriate from a dropdown - and customize/tweak/create perspectives yourself if you are not happy with the auto-generated starting point.

Importing data into Neo4j - the spreadsheet way

I am sure that many of you are very technical people, very knowledgeable about all things Java, Dr. Who and many other things - but I in case you have ever met me, you would probably have noticed that I am not. And I don’t want to be. I love technology, but have never had the talent, inclination or education to program - so I don’t. But I still want to get data into Neo4j - so how do I do that?

There are many technical tools out there (definitely look here, here and here, but I needed something simple. So my friend and colleague Michael Hunger came to the rescue, and offered some help to create a spreadsheet to import into Neo4j.

You will find the spreadsheet here, and you will find two components:

1. an instruction sheet. I will get to that later.
2. a data import sheet. Let’s look at that first.

The Data Import Sheet

This sheet is composed of two parts:

• columns A, B and C: these contain the data for the Nodes of our graph, using an “id”, a “name”, and a “type”
• columns F, G and H: these contain the data for the Relationships of our graph, having a “from-id” (where the relationship starts), a “to-id” (where the relationship ends), and a “relationship type”. Columns F and G reference the nodes and their id’s in column A.

And then comes the seccret sauce: how to create Cypher statements from these nodes and relationships. For this we use very simple statements that leverage the columns mentioned above, the cypher syntax and string concatenation. Look at the columns D and I:

• cypher statements to create the nodes:

="create n={id:'"&A2&"', name:'"&B2&"', type:'"&C2&"'};"


output for row 2:


create n={id:'1', name:'Amada Emory', type:'Female'};

As you can see, it takes that id, name and type properties from columns A, B and C, and puts these into a “create” cypher statement.

• cypher statements to create the relationships:

="start n1=node:node_auto_index(id='"&F2&"'), n2=node:node_auto_index(id='"&G2&"')  create n1-[:"&H2&"]->n2;"

output for row 2:


start n1=node:node_auto_index(id='1'), n2=node:node_auto_index(id='11') create n1-[:MOTHER_OF]->n2;

This one is a little bit more complicated, as it will be using Neo4j’s auto-index: in order to create the relationship, we first have to look up start node and end node from the auto-index using the ID property. And then the create-statement creates the relationship based on the relationship-type in column H.

So with this, we end up with two columns containing a bunch of cypher statements. So then what?

The Instructions Sheet

In the first sheet of the spreadsheet, you will find a bunch of instructions. Basically, you need to go through the following steps:

• download and unzip Neo4j server.
• copy/paste the cypher statements from the Import Sheet into a text file.
• wrap these with a neo4j transaction (begin, commit) - so that all of the statements get persisted to disk in the same transaction (or not in case of an error). Not important for smaller datasets, more important for larger datasets.
• some instructions on how to enable auto-indexing on Neo4j. This is important, because as you insert data into the database, it needs to get indexed for setting up the relationships properly (see above), and future use.

• and some instructions on how you can pipe the text file into the neo4j shell - if necessary. For small datasets (and therefore, a limited number of cypher statements) you can do with copy/pasting the textfile into the Web-UI console - but that might not always work.
• starting the server and browsing the Web-UI

And there we go: the dataset gets created, and Neo4j is ready for use. I hope this little overview was useful for you - it sure was useful for me when getting my hands dirty for the first time :) …

Podcast Interview with Andrew Bowman, Neo Technology

BY FAR the most annoying thing about working for Neo4j, is that there are so many, MANY cool things to do. And that means that sometimes cool things fall through the crack. Like for example this podcast episode, which dates from March already - a great conversation with Andrew Bowman about his work in the Neo4j community. As it so happens, Andrew just recently joined our "Customer Success" team, and is now not just an active community member - but he can actually live and breathe Neo4j 24/7 now :)) ... Here's our chat:

Here's the transcript of our conversation:

RVB: 00:03.249 Hello, everyone. My name is Rik, Rik Van Bruggen from Neo Technology, and here I am again, recording another podcast for the Graphistania podcast, and this time I've got another introduction of my dear friend Michael Hunger on the other side of this Skype call and that's Andrew Bowman. Hi, Andrew.

Loading the Belgian Corporate Registry into Neo4j - part 3

In this third part of the blogposts around the Belgian Corporate registry, we're going to get some REAL success. After all the trouble in part 1 (with LoadCSV) and part 2 (with lots of smaller CSV files, bash and python scripts) that we had before, we're now going to get somewhere.

The thing is, that after having split the files into smaller chunks and iterating over them with Python - I still was not getting the performance I needed. Why o why is that? I looked at the profile of one of the problematic load scripts, and saw this:

I checked all of my setup multiple times, read and re-read Michael Hunger's fantastic Load CSV summary, and still was hitting problems that I should not be hitting. This is where I started looking at the query plan in more detail, and spotted the "Problem with Eager". I remembered reading one of Mark Needham's blogposts about "avoiding the Eager", and not fully understanding it as usual - but realizing that this must be what is causing the trouble. Let's drill into this a little more.

Trying to understand the "Eager Operation"

I had read about this before, but did not really understand it until Andres explained it to me again: in all normal operations, Cypher loads data lazily. See for example this page in the manual - it basically just loads as little as possible into memory when doing an operation. This laziness is usually a really good thing. But it can get you into a lot of trouble as well - as Michael explained it to me:

"Cypher tries to honor the contract that the different operations within a statement are not affecting each other. Otherwise you might up with non-deterministic behavior or endless loops. Imagine a statement like this: 

MATCH (n:Foo) WHERE n.value > 100 CREATE (m:Foo {m.value = n.value + 100}); 

If the two statements would not be isolated, then each node the CREATE generates would cause the MATCH to match again etc. an endless loop. That's why in such cases, Cypher eagerly runs all MATCH statements to exhaustion so that all the intermediate results are accumulated and kept (in memory). 

Usually with most operations that's not an issue as we mostly match only a few hundred thousand elements max. With data imports using LOAD CSV, however,  this operation will pull in ALL the rows of the CSV (which might be millions), execute all operations eagerly (which might be millions of creates/merges/matches) and also keeps the intermediate results in memory to feed the next operations in line. This also disables PERIODIC COMMIT effectively because when we get to the end of the statement execution all create operations will already have happened and the gigantic tx-state has accumulated."

So that's what's going on my load csv queries. MATCH/MERGE/CREATE caused an eager pipe to be added to the execution plan, and it effectively disables the batching of my operations "using periodic commit".  Apparently quite a few users run into this issue even with seemingly simple LOAD CSV statements. Very often you can avoid it, but sometimes you can't."

Try something different: neo4j-shell-tools

So I was wondering if there were any other ways to avoid eager, or if there would be any way for the individual cypher statement to "touch" less of the graph. That's when I thought back to a couple of years back, when we did not have an easy and convenient tool like LOAD CSV yet. In those early days of import (it's actually hard to believe that this is just a few years back - man have we made a lot of progress since that time!!!) we used completely different tools. One of those tools were basically a plugin into the neo4j-shell, called the, neo4j-shell-tools.

These tools still offer a lot of functionality that is terribly useful at times - among which a cypher-based import command, the import-cypher command. Similar to LOAD CSV, the command has a batching option, that will "execute each statement individually (per csv-line) and then batch statements on the outside so they (unintentionally, because they were written long before load csv) they circumvent the eager problem by only having one row of input per execution". Nice - so this could actually solve it! Exciting.

So then I spent about 30 mins rewriting the load csv statements as shell-tools commands. Here's an example:

//connect the Establishments to the addresses 

import-cypher -i /<path>/sourcecsv/address.csv -b 10000 -d , -q with distinct toUpper({Zipcode}) as Zipcode, toUpper({StreetNL}) as StreetNL, toUpper({HouseNumber}) as HouseNumber, {EntityNumber} as EntityNumber match (e:Establishment {EstablishmentNumber: EntityNumber}), (street:Street {name: StreetNL, zip:Zipcode})<-[:PART_OF]-(h:HouseNumber {houseNumber: HouseNumber}) create (e)-[:HAS_ADDRESS]->(h);

In this command the -i indicates the source file, -b the REAL batch size of the outside commit, -d the delimiter, and finally -q the fact that the source file is quoted. Executing this in the shell was dead easy of course, and immediately also provides nice feedback of the progress:

Just a few minutes later, everything was processed.

So this allowed us to quickly and conveniently execute all of the import statements in one convenient go. Once we had connected all the Enterprises and Establishments to the addresses, the model looks like this:

So then all that is left it to do was to connect Enterprises and Establishments to the activities:

The total import time of this entire dataset - on my Macbook Air 11 was about 3 hours - without any hickups whatsoever.

So that was a very interesting experience. Had to try lots of different approaches - but managed to get the job done.

As with the previous parts of this blog series, you can find all of the scripts etc on this gist.

In the last section of this series, we will try to summarize our lessons learnt. In any case I hope this has been a learning experience for you as well as it was for me.

Cheers

Rik

My Graph Journey - part 2

In a previous blogpost, I told you the story of how I decided to get involved in our wonderful Neo4j community. I refused to make a difference between the *commercial* aspects of the Neo4j project, and the pure, free-as-in-speech-and-as-in-beer open source project and community. I believe that are one, have to be one. But. Once I had sort of made up my mind about getting stuck into it, there was a whole new challenge waiting for me. Neo4j - at least two+ years ago when my journey started, was not the easiest tool to use. There were many obstacles along the way - and while many of them have been resolved along the way, some still remain. Let me take you through THAT part of my journey - the part where I actually need to make Neo4j my friend.

I am not a programmer

Probably the single most obstacle to myself getting involved with Neo4j as a user, was that I don’t know how to program. I mean, at University I did *some* programming, but I think the world should be thankful for the fact that none of my code ever made it into production. Seriously. I suck at programming. Probably because I don’t really enjoy DOING it. I like talking about it, I love watching OTHER people do it (!), but I just don’t have the talent or the inclination to really do development. Sorry.

But let’s face it, Neo4j in 2012 was really very much a *developer tool*. It was not, by any means, something that you could hand of to a business user, let alone a database administrator, to really use in production. And I am neither of those. I am a sales person, and I love my job with a passion.
So how could I ever get stuck in with a development centric open source project like Neo4j? Well, I believe it’s really simple.

• Ask great people for help. Don’t be afraid or ashamed to say that you don’t know something, and ask the people that do know for assistance. There are some great people in our community, and even more so at Neo Technology. As one of my colleagues put it: “NeoTech is so great, because there are no assholes here…”. Haha. There’s a lot of truth in that: my colleagues are great, and they help me whenever they can. I would love have been able to write this blog, write the book, speak at conferences, without their support. 
• Failure is good. I think that’s probably the biggest thing that I learned along the way - and that I see lots of people NOT doing - is that they hold back, for fear of failure. They are standing on the sea shore, and are afraid to jump in - in spite of the fact that there are swimming teachers, rescue vests, lots of other swimmers and even the rock-star shark fighters available if something would go wrong. People just don’t try. And when they fail, they don’t ask for help (see above) and retry.

Trying something, failing, and then being able to humbly ask for help and assistance is the most powerful thing. You’re not failing because you are stupid. You’re bound to fail if you try something new… no guts, no glory! But so many people, so so many of them, never do try. It’s a shame. That’s basically how I got to try Neo4j, bump my head against brick walls time and time again, but after a while, feel like I was getting somewhere. That was a gradual process - but it felt and feels great. Now let me tell you about the two three powerful learning experiences that I had, from a more technical perspective.

Learning Neo4j

Of course, a Graph Database like Neo4j is new, or at least newish technology. So it is bound to be a bit different, and rough around the edges. If you can’t live with that, times are going to get rough. So what were the key new things that I had to get my mind around? Let’s go through the top three.

1. Learning how to Model

Modelling in a graph database is different, especially if you come from a relational background. Relational databases have many good things about them, but one of the inherent limitations to that model is that it’s actually quite “anti-relational”. What I mean is: every time you you introduce a new connection between two entities, you pay the price of having to join these two entities together at query time. Even worse in n-to-m connections, as that introduces the unnecessarily complex concept of a “join table”. So, so annoying. But the thing is, that we are used to thinking in that way - that’s how we were educated and trained, that’s how we practiced our profession for decades, so … we almost can’t help it but doing it that way.

The fundamental difference in a graph model, I believe, is that introducing relationships/connections is cheap - and that we should leverage that. We can normalise further, we can introduce new concepts in the graph that we otherwise forget, we can build redundancy into our data model, and so on and so on. I won’t go into the details of Graph Database modelling here, but suffice to say that it’s different, and that I had to go through a learning curve that I would imagine would required for most people. It pays to model - and you should take your time to learn it, or ask for assistance to see if it makes good sense or not.

2. Learning Import

Once you have a model, you probably want to import some data into it. That, for me, was probably the biggest hurdle that I had to get over in order to learn Neo4j. I remember messing about with Gephi and Talend trying to generate a Neo4j database just to avoid having to use the import tools that were available 2.5 years ago, and asking myself why oh why is that so difficult. Surely there must be better ways to do that.
I meanwhile believe that Importing data into a Graph Database is *always* going to be a bit tricky (for the simple reason that you have to write data AND structure at the same time), but that there are specific tools around for specific import use cases. Now, luckily, these tools have moved on considerably, and I think if you look at my last “summary” of the state of Neo4j import tools, it has gotten a LOT better. My rule of thumb these days is that

• for anything smaller than a couple of thousand nodes/relationships, I will use cypher statements (often generated with a spreadsheet, indeed) to import data. 
• for anything up to a couple hundred thousand, and lower millions of nodes and relationships, I will usually resort to using LoadCSV, the native ETL capability of Cypher.
• for anything that requires higher millions or billions of nodes and relationships to be imported, I will use the offline, batch-oriented tools.

It took me a while to understand that you actually need to use different tools for different import scenarios - but that’s just the way it is, at least today.

3. Learning Cypher

Last but not least, I really feel that learning Cypher, the declarative query language of Neo4j, is totally worth the while. It may seem counterintuitive at first: why do I need to learn yet-another-query-language to deal with this Neo4j thing - until you start using it. Things that are terribly hard in SQL, become trivially easy in Cypher. Queries of a 1000 lines or more in SQL, fit on half a page in Cypher. It’s just so, so powerful. And I have found that the learning curve - even for a non-developer like myself - is very, very doable. I would not call myself a Cypher expert, but I definitely feel more than confident enough today to handle quite sophisticated queries. And again: if I get stuck, I nowadays have books about Cypher, websites like Wes’, and friendly people everywhere to help me. Cypher - in my opinion - is the way to go, and Neo4j is only going to make it better with time.

That’s about it, in terms of my big lessons learnt on this wonderful Graph Journey. So let’s wrap it up.

Having fun while learning

I think the final thing here that I would like to add is that Learning Neo4j, even though a bit painful sometimes, has been a tremendously FUN experience, above all. Why otherwise would I come up with Graph Karaoke?


I believe that to be really, really important. Learning should be fun. So the more you can play with interesting datasets, the more you have the opportunity to share and discuss about that with your friends and colleagues, the more fun you will have and the more you will enjoy getting stuck in and learn some more. So set yourself up that way. Don’t be a lonely document out there - but connect with others and leverage the graph. I for one, am not regretting it for a second.

Hope this story was useful. Comments and questions always more than welcome.

Cheers

Rik