Part 4/4: Playing with the Carrefour shopping receipts

(Note: this post is part of a series - Part 1, Part 2, Part 3, Part 4 are all published!)

Alright here goes part 4 of 4 of my work on the Carrefour shopping receipts dataset. I realize we have come quite a way - and for me too there has been a lot to talk about and explore in these blogposts. Even then I feel like there's a ton of other interesting questions that we could ask and answer - but that would lead us too far.
Just to recap:

• In part 1 I wrangled the data, and imported it into Neo4j.
• In part 2 I was doing some simple but interesting queries on the data, just to get our feet wet and get a feel for the dataset.
• In part 3 we started doing some more interesting work - specifically around product combinations.

Now, in this final part of this series, I want to see if we can do some more analytical work with this dataset, for example by applying some algorithms to it. More specifically, I want to use some of our graph similarity algorithms to figure out which products are supposedly similar to one another - and do that along multiple axes. 

People have written long and complicated doctorates about the best way to calculate and establish similarities in graphs - and most of it is very much beyond me and my reptile math brain. But one thing is clear: many of the algorithms have very different approaches to doing this, and there are good reasons for wanting to choose or abandon one or the other. However, in our daily Neo4j work, we have seen some particularly interesting results with the Jaccard similarity algorithm, which is part of the algos plugin to Neo4j.

Jaccard similarity

The simple explanation of what Jaccard similarity does, is that it calculates a coefficient that compares members of two sets to see which members are shared and which are very different. So it's a measure of similarity for two sets of data - with a range from 0% (not similar at all) to 100% (identical). Higher scores mean higher similarity between the two populations. Jaccard similarity is sometimes referred to as "Intersection over Union", as explained like this:

I borrowed most of this explanation from the inevitable Wikipedia of course. You can find the Neo4j algo library that contains this algorithm over here.

Part 3/4: Playing with the Carrefour shopping receipts

(Note: this post is part of a series - Part 1, Part 2, Part 3, Part 4 are all published!)

Alright here goes part 3 of 4 of my work on the Carrefour shopping receipts dataset.

In part 1 I wrangled the data, and imported it into Neo4j. In part 2 I was doing some simple but interesting queries on the data, just to get our feet wet and get a feel for the dataset. Now in this article I want to do some more interesting work - specifically around product combinations. Which products are being bought together? Who is buying which combinations together? You can just sense that this would be some interesting stuff.

And I must say that this was quite an interesting "assignment". Originally, I wanted to actually look at all the combinations of products that we found in our dataset, and I wrote a nice little query for it:

//PLEASE DON'T RUN THIS QUERY!!!
call apoc.periodic.iterate("
match (p1:Product)<-[:TICKET_HAS_PRODUCT]-(t:Ticket)-[:TICKET_HAS_PRODUCT]->(p2:Product)
where id(p1)>id(p2)
return p1, p2","
merge (pc:ProductCombo {combo: p1.description+ ' with '+ p2.description, product1: p1.description, product2: p2.description})
on create set pc.frequency = 1
on match set pc.frequency = pc.frequency + 1
",
{batchsize:50000, iterateList: true, parallel: false})

In theory, this works just fine - and the db starts churning away and writing back ProductCombo nodes - but it never finishes. Or maybe I lost my patience :) ... but then I realised that the math is very much working against me: I have 53588 products in this dataset. If I remember my maths correctly, that means that

nCr = n(n - 1)(n - 2) ... (n - r + 1)/r! = n! / r!(n - r)!

I would have 53588! / (2! * 53586!) = 1435810078 combinations of products possible. See the StatTrek website for the calculator :) ... on top of that I realised that ALL of these combinations are probably not that interesting for us - maybe we should try to make this a bit more specific?

Part 2/4: Playing with the Carrefour shopping receipts

(Note: this post is part of a series - Part 1, Part 2, Part 3, Part 4 are all published!)

In the previous article in this series, we had started to play around with the Carrefour shopping receipts dataset that I found from a hackathon in 2016. It's a pretty cool dataset, and with some text wizardry and some Neo4j procedures, we quickly had a running database of Tickets, TicketItems, Clients, Malls and Products. The model looks like this:

In summary, we have

• about 585k shopping tickets in the dataset, 
• that hold about 6.8M ticketitems (so 11-12 ticketitems/ticket, on average)
• from 2 different Carrefour malls, 
• from 66k different Carrefour clients
• with about 53k different products

This clearly is not "big data" yet, but it's big enough to be interesting and to have a bit of a meaningful play with. So let's run some queries!

Exploring new datatypes in Neo4j 3.4 and the Open Beer Database - part 2/2

In the previous blogpost I imported the Open Beer Database into Neo4j and added some new fancy spatial data to it. Now in this post I would like to explore that data. As a reminder, you can find the full

• load script over here, and 
• some query examples over here.

Let's take a look.

First we will just look at the basic OpenBeerDB data. The schema is quite straightforward:

The Great Olympian Graph - part 3/3

In part 1 of this blogpost series, we created and prepared a dataset with all the modern Olympian medallists from 1896 to 2012. In part 2, we loaded all that data into Neo4j: here's that article. Now, we of course want to do some interesting queries on the dataset, and see if the Graph will yield any interesting insights - as it usually does.

Easy querying - number of sports per Olympic game

Let's start with something easy - doing some counting of the numbers of sports in every game since 1896. Here's how we do that:

//number of sports per game 

match (y:Year)<--(e:Event)-->(d:Discipline)-->(s:Sport) 

with distinct y.name as game, s.name as sport 

return game, count(sport)order by game ASC

Then you can see that the number of sports has not really changed that dramatically over the years: In the early days we immediately went from 9 sports at the first Olympic game in 1896, to 19 in 1900.

Graphing the Tour de France - part 3/3

In the past two blogposts I have been creating and importing some nice Tour de France 2016 data. It's a small dataset, for sure, and this is by no means a realistic graph application - but perhaps we can still have some fun exploiting the data with some cypher queries. That's what we'll try now. I have put all of the example queries together in this gist, so please feel free to play around with it :) ... let's take you through it.

Is the model really there?

First and foremost, let's verify the model that we wanted to put in place, with yet another AAPOC (Awesome APOC). We thought we were going to get this model:

Podcast Interview with Oskar Hane, Neo Technology

Here's another great conversation with one of the Neo4j Engineers that is working on the coolest product since sliced bread: me and Oskar had a great conversation about his daily work on the Neo4j Browser - one of the most loved components of a very loved product. Here's out chat for you:


Here's the transcript of our conversation:

RVB: 00:02 Hello everybody. My name Rik, Rik Van Bruggen, from Neo Technology. Here we are recording another podcast session. Today, we have a really interesting guest, all the way from Borås in Sweden, and it's one of our developers on the Neo4j engineering team. It's Oskar Hane. Hi Oskar.

OH: 00:20 Hi Rik.

RVB: 00:21 Hey, good to have you on the podcast. Thanks for making the time.

OH: 00:25 Thanks for having me.

RVB: 00:27 It's great. Oskar, one of the reasons I was really excited to get you on the podcast was because Neo4j has this wonderful front-end tool that I use a lot, everyone uses it a lot, called the Neo4j browser. I think you're one of the guys behind that, aren't you?

OH: 00:46 Yes, since about one and a half years, I've been on that UX team, what we call it internally.

RVB: 00:54 How did you get to Neo, who are you, where did you come from, how did you get to Neo?

OH: 01:01 Actually, it's quite a funny story, I think. I was browsing Twitter two years ago, and I think that one of the guys I was following retweeted something from a guy called Emil Eifrem, send that--

RVB: 01:21 Who is that guy?

OH: 01:22 By then, I didn't know. I didn't know [anything?] at all, and he said that, "We're looking for Java developers." I was a freelance contractor by then, and I was just-- a long contract, so I was looking for Neo4j, so I sent him a tweet and said, "Don't you have any openings for JavaScript because I'm not a Java guy?" And he was like, "I think we do, actually. Talk to Magnus!", another guy inside Neo. They set up interviews. I actually have like seven or eight interviews before I joined in August of that year.

RVB: 02:16 Have you used Neo4j before or did you know about the product before, or how did you know about it?

OH: 02:22 No, I didn't, but of course, when I talked to Magnus the first time-- well, before I talked to Magnus, I downloaded and started using it. I was blown away by the user interface of the client or the Neo4j Browser compared to-- I've been working with MySQL and other databases, and their user interface--

RVB: 02:53 Is a little bit more old-fashioned?

OH: 02:55 Yeah, so I was completely blown away. I'm thinking, "Wow, I'd really like to work on this product," and I managed to get in.

RVB: 03:07 And the rest is history, right?

OH: 03:09 Yeah [chuckles].

RVB: 03:10 Nowadays, it's you and one or two other people that work on that part of Neo4j, right?

OH: 03:16 Yeah, currently, we're two. The other guy, Mark Peace sits in the London office, and in two weeks, another person is joining us, so it'll be great to have the three of us.

RVB: 03:33 What do you think is so cool about Neo4j, in general, but maybe the browser, more specifically, why do you think it's such a cool product to work on, what makes it so nice?

OH: 03:47 As you were talking a little bit that it's modern, in a way, compared to MySQL, and the browser is, you can visualize the data in a way that you're not able to visualize in a relational database, of course, an area that Neo4j or graph databases we have fixed relationships between nodes.

RVB: 04:23 The graph data model, right?

OH: 04:25 Yeah, exactly. So we have this graph visualisation in the Neo4j browser where we can run animations and have a gravity physics engine to display the data in a nice way.

RVB: 04:44 I think one of the cool things about the browser, for me as a user, is that it combines so many nice perspectives. On the one hand, it's a visualisation tool, on the other hand, it's a query troubleshooting tool, on the other hand, I have a learning tool. There's so many nice things coming together, I think. I'm assuming that was intentional.

OH: 05:09 Yeah, of course, it's sort of a platform for-- the target audience was from the beginning, at least, for developers who are writing apps for the Neo4j database, then we can use this Neo4j browser, as you say, they can see the data in many different ways. We can troubleshoot if we have any query program, so we can just run ad hoc queries to get interesting stuff, and you can learn the query language, Cypher, within the browser.

RVB: 05:50 I have off the cuff type of question for you, what's your favorite feature, if you have any? It's probably the most recent one [chuckles], but what is one of your most-- the nicest thing about the browser?

OH: 06:08 Wow, that's really a hard question. Even though it could be limited in how big graph you can do it at once, but I think the visualisation is actually my favorite because I'm basing that element that other database types that don't have, that makes us unique.

RVB: 06:37 My personal, probably, is the fact that you have this rolling history. You have this long page of query histories that you can sort of go through. I don't know if you've ever seen one of my graph karaokes but that's what I use all the time...

OH: 06:54 I have that actually, I have that karaoke [laughter].

RVB: 06:59 I'm a fan of your work, Oskar, I really am.

OH: 07:03 Thanks.

RVB: 07:05 One more thing, if you don't mind, where is this going? What does the future hold, both from the product in general, the browser, more specifically, and maybe the industry that we work in, any comments on that?

OH: 07:23 I can talk for the browser at least that we-- we're talking about having it extensible in a way that somehow that you can write like from the plugins maybe for the browser, and load in an easy way as a user. I very much like to see that, at least, that if you want to visualize your data in a special way, maybe your own [row?] or table format that only you see, you could create a front-end plugin, so to speak, just have it that way.

RVB: 08:10 So like an extension, but then part of the browser, so to speak, right?

OH: 08:14 Yeah, something like that. But we're moving some of the content out of the browser so you can bring it back in. We're heading that way.

RVB: 08:27 You also mentioned to me earlier that for future versions of Neo4j, the browser is not going to be like part of the same package as Neo4j, is that also something that's coming up then?

OH: 08:37 Well, maybe, I'm not 100% sure where we are at the decisions yet, but what we know is that, at least, we have among the files, the repository for the browser out of the Neo4j main database, but mostly, having it as a dependency. So when you download Neo4j, you get the browser inside it, just as you do now, but the code isn't mixed with the database code.

RVB: 09:08 It's easier to pull it out if we want or need to do that, right?

OH: 09:12 Yeah, exactly.

RVB: 09:13 Very cool. What about visualization, in general, is there-- there's so much stuff happening in that domain, graph visualization, it's a very hot topic, I think, how do you look at that?

OH: 09:28 Yeah, it is, and it's super hard to show visualization that are huge which has functions and elements and relationships within it. I'm very impressed by many of the actors in the industry.

RVB: 09:51 It very quickly becomes like hairball, doesn't it [chuckles]?

OH: 09:56 It can hold your web browser as well.

RVB: 10:00 Totally, there's some really cool visualization tools out there. We've been partnering with externally, so hopefully, that's complimentary, it should help us.

OH: 10:13 Definitely, we're a small team. I think our visualisation is good for some use cases, but maybe not for some.

RVB: 10:26 Well, very good. We'll wrap up the recording here, if you don't mind. We'd like to keep this podcast short and snappy. Thank you so much for coming online, Oskar, really appreciate it.

OH: 10:40 Thanks.

RVB: 10:41 I'll look forward to seeing you maybe at GraphConnect Europe or some other occasion very soon.

OH: 10:47 I definitely will be at GraphConnect Europe.

RVB: 10:50 Fantastic. I'll look forward to seeing you there. Thank you.

OH: 10:52 Likewise, thank you.

RVB: 10:53 Bye.

Subscribing to the podcast is easy: just add the rss feed or add us in iTunes! Hope you'll enjoy it!

All the best

Rik

The GraphBlogGraph: 3rd blogpost out of 3

Querying the GraphBlogGraph

After having created the GraphBlogGraph in a Google Spreadsheet in part 1, and having imported it into Neo4j in part 2, we can now start having some fun and analysing and querying that dataset. There are obviously a lot of things we could do here, but in this final blog post I am just going to explore some initial things that I am sure you could then elaborate and extend upon.

Let’s start with a simple query

// Which pages have the most links
match (b:Blog)--(p:Page)-[r:LINKS_TO]->(p2:Page)

return b.name, p.title, count(r)

order by count(r) desc

Run this in the Neo4j browser and we get:

or just return the graphical result with a slightly different query:

match (b:Blog)--(p:Page)-[r:LINKS_TO]->(p2:Page)

with b,p,r,p2, count(r) as count

order by count DESC

limit 50

return b,p,r,p2

And then you start to see that Max De Marzi is actually the “king of linking”: he links his pages to other web pages a lot (which is actually very good for search-engine-optimization) .

A quick visit to one of Max’ pages does actually confirm that: there’s a lot of cool, bizarre, but always interesting links on Max’ blogposts:

So let’s do another query. Let’s look at the different links that exist between blogposts of our blog-authors. Are they actually quoting/referring to one another or not? Let’s do

//links between blogposts

MATCH p=((n1:Blog)--(p1:Page)-[:LINKS_TO]-(p2:Page)--(b2:Blog))

RETURN p;

and then we actually find that there are some links - but not that many.

Same thing if we look at this a different way: let’s do some pathfinding and check out the paths between different blogs, for example my blog and Michael’s

match (b1:Blog {name:"Bruggen"}),(b3:Blog {name:"JEXP Blog"}),

p2 = allshortestpaths((b1)-[*]-(b3))

return p2 as paths

Then we actually see a bit more interesting connections: we don’t refer to one another directly very often, but we both refer to the same pages - and those pages become the links between our blogs. At depth 4 we see these kinds of patterns:

Interesting, right? I think so, at least!

Then let’s do some more playing around, looking at the most linked to pages:

//Which pages are being linked to most

match ()-[r:LINKS_TO]->(p:Page)

return p.url, count(r)

order by count(r) DESC

limit 10;

That quickly uncovers the true “spider in the web”, my friend, colleague and graphista-extraordinaire: Michael Hunger:

Last but not least, I wanted to revisit an old and interesting way of running PageRank on Neo4j using Cypher (not using the Graphaware NodeRank module, therefore). I blogged about some time ago, and it’s actually really interesting and easy to do. Here’s the query:

UNWIND range(1,50) AS round
MATCH (n:Page)
WHERE rand() < 0.1
MATCH (n:Page)-[:LINKS_TO*..10]->(m:Page)
SET m.rank = coalesce(m.rank,0) + 1

This does 50 iterations of PageRank, using a 0,1 damping factor and a maximum depth of 10. Running it is surprisingly quick:

If you do that a couple of times, and even do a few hundred iterations at once, you will quickly see the results emerge with the following simple query:

match (n:Page)
where n.rank is not null
return n.url, n.rank
order by n.rank desc
limit 10;

Confirming the “spider in the web” theory that I mentioned above. Michael rules the links!

All of these queries are of course on Github for you to play around with. Would love to hear your thoughts on these three blogposts, and hope that they were as fun for you to read as they were for me to write.

All the best.

Rik