Here's the transcript of our conversation:
RVB: 00:02 Hello everyone. I'm name's Rik, Rik Van Bruggen, from Neo Technology and I'm at the London coffee shop of choice near our office in London doing another interview for the Neo4j podcast and I'm interviewing someone who I have interviewed before [chuckles] a while ago - Nigel Small from our engineering team. Hi, Nigel.
NS: 00:27 Hello.
RVB: 00:28 Hey, it's good to have you here. The reason for inviting you, somewhat unexpectedly, to talk a little bit on this podcast interview is that I know you've been hard at work for the past, I don't know, 18 months or so on the Bolt interface to Neo4j and the new drivers for the different development languages. I wanted to talk a little bit about that. Can you tell us a little bit about what is Bolt and what are the new uniform drivers? Let's start there.
NS: 01:01 So Bolt is the new approach for interfacing with Neo4j to eventuate probably replace the old REST interface, but for now, sits side by side it. It's a binary protocol, it's developed entirely in-house, and we've built a set of four drivers initially that we've released that interface with Java, Python, Java Script and .Net, to try to broaden the reach of the software that we're offering in-house and offer something that's supported for those platforms.
RVB: 01:37 Absolutely. So Bolt, from what I understand it, it's a binary protocol as opposed to the REST interface which is ASCII-based, I suppose. It's clear text.
NS: 01:49 It's text based. So the old interface - the HTTP interface - using JSON to transmit its payloads and as a data transfer format, JSON has challenges, let's say. There's certain limitations in what you can express. So we've developed a custom-serialization format and it's very much in line with the Cypher type system. It's inspired heavily, let's say, by MessagePack, which is a similar set-up, but it was developed from scratch to, as I say, work with a type system and work efficiently in the way we want to transfer data to and fro.
RVB: 02:34 So what was the primary goal of Bolt? What are some of the main advantages of using it with Neo4j 3.x going forward?
NS: 02:46 As I say, the type system is certainly one of them, so you get a much more native type system. You're sending fewer bytes to and fro, and while we haven't focused very heavily on optimization at this stage, we want to get the features set, fully fleshed out, there's a lot of optimization ideas we have going forward. Already, generally speaking, we're going to have a much faster experience with Bolt than you have done with HTTP in the past.
RVB: 03:13 As I understand it, it makes the server mode of Neo4j, as opposed to the embedded mode of Neo4j, a lot more feasible for high performance applications, or is that not really the case?
NS: 03:29 No, there are certain advantages. You've got a stateful session that you know you use, as opposed to HTTP which uses a stateless set-up. So each time you make a request when using HTTP, you're sending often the same set of metadata across, you're sending your user agent, you're sending your authentication information with each request that you do. With Bolt, you send that information at the start of a session and then that's used throughout, so you don't need to resend the same data. So, yeah, you do get some efficiencies.
RVB: 04:03 Very cool. But as I understand it, some of the side-effects from implementing the Bolt protocol, that hasn't been more on the drivers side? There has been a lot of work that you've been doing also on the uniform drivers that we've been providing with three.org. Can you talk to us a little bit about that?
NS: 04:23 Yes. The uniformity has been a key part of this. We wanted to provide a clean uniform experience across different languages. We picked these four to start with because they were four of the most important ones to us.
RVB: 04:37 Which four are those?
NS: 04:37 The four are: Java, Java Script, Python and .Net. And we wanted to make the experience as similar as possible. So we've made sure that we've unified the use of terminology and concepts that the drivers use across the board. So we have a session, we have a transaction, we have a results set. And they're all handled and described in the same way across all the drivers. In fact, the developer manual - the new developer manual - actually has one story it tells of how to use the driver with simply a difference in the sample code that's embedded. You can just switch the tab and it shows you the same code in different languages.
NS: 05:19 So we wanted to get this uniformity in place, but a lot of the difficulty has been around making sure that we get the right balance between uniformity and idiomatic language use. So we didn't want something that was exactly the same in every language, but for alien to the developer in that language. And actually getting that balance right has been quite a lot of work, it's been a real challenge at times, and we've tried to respond to feedback where developers have come to us. And we had a particular incidence with that with the .Net users telling us that the methods we were using for filtrating through results didn't feel natural. So we've gone back and we've reassessed how we were doing that - this was still pre-release of 3.0 - and we went back and we reassessed and we redesigned and we actually shifted the balance there much more towards idiomatic and away from uniform. I think that's been something we couldn't do entirely in-house, we needed to talk to the users in those ecosystems and tell us how best that we could fit that in. I think now we've got something that's pretty solid and should work well in most languages.
RVB: 06:32 In the past, the language drivers around the REST API in the past, they were mostly developed by the community, right? They were mostly developed by people like yourself with other people from the community have been contributing. Has that changed as well now with the uniform driver set?
NS: 06:50 We still have some driver authors. This is interesting actually. I've been involved with doing this for about five years or so now in terms of developing drivers, and seeing some drivers that have been born and had a life and then died off somewhere, other drivers that have carried on. But now we have some official drivers, actually we have to kind of work out how we want those drivers to sit alongside the community efforts. We don't want to go along and rid ourselves of any community efforts we have because the community's very very valuable and we don't want to build all these comprehensive idiomatic features in every single language. We want to provide a base, I think, is where we've left this. We want to build a base core driver that handles all the plumbing, doesn't let you have to worry about the type system and the protocol detail, provides a base API on which you can build other layers, build an OGM, build other things that are specific to the language that you're in. You've got Link and .Net, the things that are specific to the language. So ultimately, we're hoping that the community drivers will be something that will actually sit alongside the official drivers and perhaps as a set of plug-ins or something that can extend the official driver.
RVB: 08:19 So then the official supported drivers will be like the infrastructure for more added features, feature reach and implementations by the community?
NS: 08:27 Absolutely. Exactly how we go about that, we haven't entirely decided yet. As I'm still running the Py2neo Project, I've got a few ideas of how we can kind of combine the official Python driver with Py2neo with the extra features that I've added in in a way that I don't have to duplicate my efforts and actually do the same thing at home that I do in the office. There are a few challenges there and how we fit that in, but we want to make sure there's room for both, I think, ultimately.
RVB: 08:56 Very cool. Well I think people can find a lot more information on Bolt and on how to write drivers and everything online. We'll include some of those links on the transcription of the podcast. I think it's a great evolution that we're really taking this forward. So thank you very much for taking the time and spending some time with me. Thank you for the coffee is what I wanted to say as well. And in the meanwhile, I think England has scored here. I heard a big row outside, so that's probably good news.
RVB: 09:31 All right. Thank you, Nigel, and I'll talk to you soon.Subscribing to the podcast is easy: just add the rss feed or add us in iTunes! Hope you'll enjoy it!
All the best
Rik
No comments:
Post a Comment