The Agility Angst

First of all: let me start by showing my age. I was born in 1973, and yes, that does mean that there's a big, and I mean BIG, party coming later this year. Big Five Oh looming around the corner - and to be honest I am very, very fine with that. I am happy with my age - except when I am being dropped like a baby at the back of the cycling bunch. Then not so much - dammit.

What does my age have to do with anything? Well - it basically means that I grew up with a very, very different type of software development practice. OO was still young. The mythical man-month was still a thing. And methodologies all depended on strict, rigid waterfall development models.

Nothing wrong with the odd waterfall, of course, but with the emergence of more and more, and better and better, software tools, libraries and frameworks - this whole idea of the time consuming waterfall process has kind of become irrelevant. Who would still go through this entire process and ask their users to wait for months before they would be able to give any feedback? What users would accept that type of arrogance on behalf of their IT department? Seriously?

The modeling mismatch

After having spent 10+ years in the wonderful world of Neo4j, I have been reflecting a bit about what it was that really attracted me personally, and many - MANY - customers as well, to the graph. And I thought back to a really nice little #soundcloud playlist that I made back in the day: I basically went through dozens of #graphistania #podcasts that I had recorded, and specifically went back to the standard question that I was asking my interviewees on the podcast: WHY do you like graphs??? WHY, for god's sake!!!

Unsurprisingly, people very often came back with the same answer: it's the DATA MODEL. The intuitive, connected, associative, visual, understandable structure that we humans love interacting with: the labeled property graph (LPG). Have a listen to what people were saying:

Graphistania · The Best Thing about Neo4j

Podcast Interview with Eddy Wong, Wanderu

I have said it before and I will say it again: the nicest thing about doing these podcasts is that I get to talk to so many cool people. Here's another one: Eddy Wong is the CTO and co-founder of Wanderu, and has been active in the Neo4j community for a very long time. As you will see/read/watch below, they have done some very cool stuff in combining Neo4j with MongoDB, and that's actually something super interesting for many users. So read on, listen, and try to enjoy it as much as I did :) ...

Here's the transcript of our conversation:

RVB: 00:02 Hello everyone. My name is Rik. Rik Van Bruggen from Neo Technology and here we are again recording another episode for the Neo4j Graph Database podcast. And today I'm very excited to have another overseas guest on this episode, that's Eddy Wong from Wanderu. Hi, Eddy. 

EW: 00:22 Hi, how are you doing? 

RVB: 00:23 I'm doing very well, and you? 

EW: 00:25 Good. 

RVB: 00:26 Excellent. Excellent. So Eddy, I've been reading all of the blogs and watching some of your videos from GraphConnect (see below) about your work with Neo4j at Wanderu. But most people won't have done that yet, so maybe you  can introduce yourself very quickly if that's okay? 

EW: 00:44 Okay. I'm a co-founder of Wanderu. Wanderu is a search engine for buses and trains, and the reason we were looking for a graph solution is we wanted to model the network of transportation so that you could route two buses, or two trains, or a bus and a train very easily. And that led us to using Neo4j. 

RVB: 01:19 Super. How long have you guys been doing that already? 

EW: 01:22 Since 2012. We were one of the early users of Neo4j. I actually learned about Neo4j that year and I attended the first ever GraphConnect conference in San Francisco. 

RVB: 01:43 Yeah, that must be it. 2012, somewhere around that time. We're doing a fourth conference, a fourth year this year I believe. So, it's going to be exciting. So, search engines for trains and buses. How is it different from things like what I do? Route planning on Google Maps and stuff like that, is it very different? 

EW: 02:04 Well, what you do with Google Maps is mostly local. We're in a city so, we at Wanderu, we focus on intercity travel. 

RVB: 02:22 Okay. Super. Route planning does seem like a real good use case for graphs. But I also notice in some of the videos and the blogs that you guys are actually combining it with document storage right? Can you tell us a little bit more about that? 

EW: 02:39 Yeah. That was kind of a unique approach that we end up taking because our-- most of our data was in JSON and we were using MongoDB to store that. It was very convenient to just dump the JSON there and it would be indexed. 

RVB: 03:04 That makes sense. 

EW: 03:06 And at that time, there was no easy way to upload data in JSON format  to Neo4j. Usually you have to use the CSVLoader or write some custom code or write cipher. It wasn't very convenient to upload JSON into Neo4j, so we came up with a unique approach of using an open source project called Mongo Connector. So this is a piece of software that lets you add a trigger to MongoDB, so whenever you add data to Mongo, you insert or delete something, Mongo automatically makes a callback. Inside that callback we populate Neo4j. 

RVB: 04:19 Do you also write directly to Neo4j or always through that connector? 

EW: 04:24 Through that connector. Yeah, always through that connector. 

RVB: 04:28 Super interesting. 

EW: 04:30 Sorry. That way we extracted the meta data. We didn't copy everything, but just the meta data that is useful for routing. 

RVB: 04:47 Yes, makes a lot of sense, right? Just the keys and stuff like that, right?
EW: 04:52 Yeah the keys and the edges. I mean the edges that the trip  access from point A to point B. 

RVB: 05:02 So how did you guys get to Neo4j? I mean what was the attraction and why did you end up using Neo specifically? Can you elaborate on that? 

EW: 05:13 Yeah. So I looked at other graph databases. And at that time, the only ones that I could find or read about were proprietary solutions. They were expensive and they were closed source and they were not-- so Neo4j was the only one that was open source and had a vibrant community at that time. 

RVB: 05:44 And was it a good fit you think? Do you still think it's a good fit? 

EW: 05:48 Yeah, definitely. 

RVB: 05:51 Super. And how successful is Wanderu these days? Are you guys making good traction? Are you attracting good communities? 

EW: 06:00 Yeah. So we've grown from zero to now we have over a million users per month. About sales, we sell several thousand tickets everyday, and the architecture since day one hasn't changed that much. So, our solution has remained scaleable. 

RVB: 06:27 That's very impressive, very cool. So, what does the future hold, Eddy? How is this going to evolve going forward, both from a graph perspective and, maybe also a little bit, where is Wanderu going? 

EW: 06:44 We like to think of ourselves as travel for the next generation, so graphs enables us to model data in a way that that hasn't been done before. So in the future-- you look at social networks and all the information that is stored in graphs. So, eventually you have a social network connecting with a transportation network, and you can imagine all the cool stuff they can do with that. 

RVB: 07:30 So things like recommendations and stuff like that, is that the type of stuff you're thinking about? 

EW: 07:35 Yeah. 

RVB: 07:37 And that's stuff that's in the pipeline already? Or is that just thoughtware? 

EW: 07:43 Well, I can't tell you about it. 

RVB: 07:47 Oh man, I was looking for a scoop there [laughter]. Okay, well that sounds super interesting. And then what about Neo4j? How do you think that's going to evolve? And what are you hoping for, or stuff that you think will be super-useful? 

EW: 08:06 Well, there's one feature that we are really hoping that happens this year is the new geospatial plug-in. That would allow us to make our results even more interesting. 

RVB: 08:30 Okay. Are you guys in GeoData already right now? 

EW: 08:33 Yes. 

RVB: 08:36 Okay, all right. But that's in Mongo these days? 

EW: 08:37 Yeah, that's in Mongo. 

RVB: 08:40 Super. Thank you for sharing your perspective. I don't know if you have any final words or final considerations for our listeners. Anything in particular? 

EW: 08:53 Yeah, that Neo is a great product, and the community is great. I mean from day one going to the first GraphConnect it was great to interact with the community. The community's very enthusiastic and it's great to interact with them. 

RVB: 09:23 Super. I'm hoping that you can make it to GraphConnect as well this year. 

EW: 09:26 Yeah. 

RVB: 09:29 Super. I'll hope to see you there then. Thank you for coming on the podcast. I really appreciate it and, when I'm in the US in October, I'll have to give Wanderu a try myself. 

EW: 09:39 All right. Definitely. 

RVB: 09:42 Thank you, Eddy. Have a nice night. Bye. 

EW: 09:45 Bye.

Subscribing to the podcast is easy: just add the rss feed or add us in iTunes! Hope you'll enjoy it!

All the best

Rik