Here's the transcript of our conversation:
RVB: 00:02 Hello, everyone. My name is Rik, Rik Van Bruggen from Neo Technology, and here I am recording a podcast episode with one of my dear British colleagues, all the way from across the Channel, Ben, Ben Butler Cole. Hi, Ben. Thanks for joining me.
BBC: 00:16 Hi, there.
RVB: 00:18 Thanks a lot for coming online. I appreciate it.
BBC: 00:20 Not at all.
RVB: 00:21 Very good. Ben, I invited you because I know you've been doing some really cool stuff on Neo4j in the engineering department, and I think it's always cool for people to sort of have a little bit of a view on how these things work. But before I get into that, why don't you introduce yourself and people might learn a little bit more about who you are and what you do at Neo?
BBC: 00:45 I'm Ben. I am one of the developers at Neo. I've been at Neo for two-and-a-half years. In that time, I spent some time working on the core product. I've also worked on improving our internal development infrastructure, build systems and release processes and so on. I've spent some time working on building testing tools to allow us to automate and improve the testing of Neo4j, particularly testing in sort of real-world scenarios, testing end-to-end in the same way that our users use it so we can have the chance to beef the product a bit before we release it to the real world.
RVB: 01:45 Absolutely.
BBC: 01:46 Most recently I started a new stream of work to improve the operations surface of Neo4j. So we're looking at logging, configuration, packaging, and things like that, trying to really improve those and trying to make the product as easy and nice to use for systems operators as it is for the end users.
RVB: 02:19 Yeah. Could you tell us a little bit more about some of these build processes and how they work? Just from a really high level. I know everything we do is open source, right, so people can look at some of the source codes on GitHub and stuff like that, but how does it actually work from a developer's perspective? What are some of the big bricks?
BBC: 02:40 Yes. We use TeamCity, which is a build server, and we take the source that's in the GitHub repo, which as you see, people can see. We built that, run a test in that. Then we have quite a long complex pipeline of builds that follow on from that. Some of the testing we do is in the public source code repository. We also have a number of internal private repositories which have testing tools and so on. For each build we probably run 10 or 15 different kinds of tests that we have that test different aspects of the application. Some very simple ones, which we call tyre kicking, which just start the application - install it, start it, run it, make sure that we can write data and read data. And then varying levels of sophistication beyond that. Low-level tests for components where we want to stress-test or performance-test an individual component, and then larger end-to-end tests. Some of the most useful tests we've got are for our testing of our clustering where we do sort of fuzzy testing, where we stand up a cluster and read and write data to it. And while it's running, we knock over individual instances - deliberately crash them or shut them down cleanly - and make sure that the cluster stays up and is resilient to that. That's been a very useful aspect to the testing that we do.
RVB: 04:30 Wow. Some of those tests, do they take a long time, or is it like instantaneous, or how does that work? Some of these things must take quite some time, no?
BBC: 04:41 They do, yes. We have tests that run for several hours for that reason, because we're standing up real clusters of service, and we want to run them over time and make sure that they're stable over a long period of time.
RVB: 04:58 Very cool, very cool. And you're now starting some new work on the operability, you said, right? There's a lot of people looking at that, I think. Some of our primary users are the administrators, aren't they?
BBC: 05:11 Yes, exactly. Historically it's an area of the product that hasn't got quite as much love as it might have done. We've been focussing on end-user features of the product, and stability and the reliability of the data storage. But we've taken a decision to make an investment in trying to improve the operability of the product as well now. So we're improving the configuration and the way that works, the packaging particularly. It's not glamorous stuff, but because of my interests I'm very excited that we're doing it. So we're changing the directory structure of the application's tools too so that it's more sympathetic towards the standard ways to doing things on the platform, particularly on the Linux, where the majority of our production installs are running.
RVB: 06:18 Is that work going to be visible in 3.0 or in the 3.x series, or...?
BBC: 06:23 Yes. Some of that work has already gone into the first steps of the milestones, have already gone into the code base and will be in 3.0.
RVB: 06:32 Super cool.
BBC: 06:33 We're trying to get as much of it as possible into 3.0 as a major release because we're happy to make some backwards incompatible changes for 3.0 because it's a major release, and then hopefully things will settle down for the minor releases that come after it.
RVB: 06:52 Totally, yeah. I always ask a couple of questions on this podcast, right? One of the things that I'm always interested in is what attracted you to Neo, and how did you get to Neo, and why do think it's a cool product to be working on. You can give me the real answers, Ben. I know there's a very boring answer to this one [laughter].
BBC: 07:15 I've spent nearly ten years before I came to Neo working as a consultant. So I saw a huge range of different systems and applications, and built quite a lot of them. One of the really attractive things that I see is what I think is the superiority of the Graph model as a way of modelling the real world and the shape of data in the real world, over SQL or key-value stores. I find that very appealing. I think we're effectively making the life easier of all those developers who I've been working with over the years, who are struggling with the impedance mismatch between particularly SQL and the applications they're trying to rhyme.
RVB: 08:18 Very cool.
BBC: 08:18 On a more personal level, I knew a bunch of people who are here working at Neo Technology, and they were people who I knew and respected, and I was keen to come and work with them. So that was what really sucked me.
RVB: 08:32 Exactly. There's a bunch of people that come from the same background at Neo, right, people like Ian and Jim and Alistair and all of those folks?
BBC: 08:38 Yeah, exactly. And they're all people who I'd worked with before, who I was keen to work with again.
RVB: 08:43 Very cool. Maybe one more question, Ben, if you don't mind. Where do you think this is going now from a product perspective, from an industry perspective? Anything that you aspire or think we should be doing or believe we should be doing, or that type of stuff? Look into the future crystal ball.
BBC: 09:09 I think there's a lot of work for me to do, carrying on the work I'm doing at the moment for the product. As you know, there are initiatives and exciting new features being built across the product at the moment, for 3.0 and beyond. I'm keen to kind of stick with the boring stuff. I really want to make Neo4j a very easy product to operate. So beyond just cleaning out what's effectively debt that we're working on at the moment, I have ambitions for monitoring particularly of live systems, to make the software able to explain to people what's going on inside it, integrate it with standard monitoring systems, and once we've reached a level of where we're happy that it's good enough, I've then got ambitions to start pushing on the state of the art, and improving on the state of the art for monitoring particularly, turning monitoring into a kind of feed of events so that it's really easy to understand, to interpret the behaviour of the system, the people operating it to be able to more or less leave it to tread away on its own, and then help build up the systems that can interact with it, and fix problems as they come up.
RVB: 10:59 Very cool. Well, thank you so much, Ben, for coming online and sharing that with us.
BBC: 11:04 Awesome.
RVB: 11:05 It's always great to get like an inside-peak in how things work in Neo's engineering world. Really appreciate it. And I would say: You know what? Let's make Neo4j boringly fantastic, right [laughter]? That would be such a great achievement. Thank you so much, Ben. I appreciate it.
BBC: 11:26 All right. Thanks.
RVB: 11:28 Cheers, man. Bye.
BBC: 11:29 Bye-bye.Subscribing to the podcast is easy: just add the rss feed or add us in iTunes! Hope you'll enjoy it!
All the best
Rik
No comments:
Post a Comment