As always, here's the transcript of our conversation:
RVB: 00:01 Hello everyone, my name is Rik, Rik Van Bruggen from Neo, and here I am recording a podcast episode together with a wonderful guest on the podcast episode, all the way from California, Ashley Sun. Hi Ashley.
AS: 00:17 Hi Rik, thanks for having me.
RVB: 00:22 Thanks for coming on the podcast. Ashley, you work for--
AS: 00:23 Of course.
RVB: 00:23 --Lending Club, right?
AS: 00:26 Yes.
RVB: 00:26 I've seen some of the videos of GraphConnect
and I've read a little bit about what you guys are doing, but why don't you introduce yourselves to our listeners?
AS: 00:37 Okay. Hi, I'm Ashley. I work on the DevOps team at Lending Club based in San Francisco. I work a lot on deployment and release automation, and I use Neo4j to do it.
RVB: 00:54 Wow, that's great. How long have you been doing work with Neo4j, Ashley? It must have been for a long time already or--
AS: 01:01 Only a little over a year, I'd say, so my manager first introduced it to me. I think he stumbled upon graph databases on Twitter or something and he's like, "Hey, check out this new thing called Neo4j." And so, we started playing around with it, and it quickly evolved from just a side project to being a really critical part of a lot of our release and deployment automation, infrastructure mapping, app auto-discovery, and a lot of other things, actually.
RVB: 01:37 That's a great segue into what do you guys use if for exactly? Why don't you tell us a little bit more about that?
AS: 01:43 Sure. So, we use if for a lot, a lot of things, actually. So, as I was saying, I guess it was very opportunistic when we started using Neo4j. We had a lot of problems in DevOps and growth pains. So, we started with maybe like five micro-services and a couple years later, we're almost at 150, and so it was getting really difficult to manage and keep track of all these services, and so we did is we use Neo4j to keep track of all these instances. We had them radio home-- we have this internal app called MacGyver. So, we had them radio home every minute to MacGyver, and MacGyver would save all these app instances in Neo4j, and so already, immediately, we just gained a lot of visibility into what services were out there, where they were running, a lot of info like that. And it was really low maintenance, it was easy to scale, we didn't have to do any work because these new instances would just keep reporting back to MacGyver and get saved into Neo4j.
So, from there, we were like, "Oh, this is really useful, so we're going to take this a step further." And so, at Lending Club, we use blue-green deployments. Basically, this just means that we have two pools for every app, and at any given time, only one pool is live. We didn't have a good way before to track what pool is live and what pool is dark, and so we started using Neo4j. We were already mapping our app check-ins, and so we took that and then within Neo4j, created the server nodes, which we then mapped to belong to pool nodes, which we then mapped to belong to service groups. By keeping track of what servers existed in what pool, and whether that pool was live or dark, we were able to automate our releases. Whereas before, releases were very manual. We'd have to go into this GUI and check mark all these boxes; it was just very tedious and very time-consuming. Now with Neo4j, we are keeping track of these info, and so it was just really quick. It was like a flip of a switch and we could make a pool live or dark or even really easily look up what pool is live. And also we used it to track the health of our instances and our apps. So, that was also really, really important and that's what we're using now for deployments.
RVB: 04:18 Super cool. By the way, I love the naming. I'm a big fan of MacGyver [laughter].
AS: 04:24 Awesome. Actually, my manager came up with the idea of the name and I had no idea what MacGyver was and just-- my time of-- I'm like, "Is that like MacGruber from SNL?" So, I had to watch an episode of MacGyver to--
RVB: 04:41 I guess I'm showing my age here a little bit [laughter]. Ashley, so it's basically what you're using for dependencies between all the micro-services, is that what I'm hearing? You're basically tracking everything with these automated ping backs, but then you're mapping it onto like a model of all your micro-services, is that what I'm hearing?
AS: 05:05 Yes, that's correct. So, taking the instances and then arranging that data in a way that becomes useful, so that we know where our apps are and what's live at any time and what's dark. And also even-- so, we're mapping with dependencies from services. So, we map them onto-- for example, vCenter instances and vCenter hosts, and then we take those vCenter arrays, then map those to our storage arrays and our storage volumes. So, what we get is like this huge mapping of our infrastructure. For example, if we want to find a single point of failure, for example, we have an app called ABC and all of its instances reside on one vCenter host, and if that host goes down, then our entire app is wiped out. That's a single point of failure. So, we use Neo4j to keep track of things like that, to avoid these-- it would be a huge disaster if that were to happen.
RVB: 06:12 Yeah. And I seem to recall from one of your talks that you also use this tooling to help you guys do more stuff with Amazon web services. Can you tell us a little bit more about that or did I get that completely wrong?
AS: 06:28 No, no. You're totally right. One of our multi-year projects is we're moving into AWS, and so we're just starting that process now, but already we are mapping a ton of AWS stuff into Neo4j. For example, like our VPCs, subnets, availability zones, our RDS instances and EC2 instances, those map to load balancers, auto-scaling groups, launch configs. You can tell already, it's like a huge, a huge map in Neo4j. So, there's all these different parts, but we are able to make sense of it by mapping relationships together in Neo4j, and also as we move into AWS, we'll start using code deploy. But again, using Neo4j to automate that and put it into MacGyver, so that developers at any time can say, "Hey, I need an instance and I want to launch this app onto it." It'll just be really simple and we'll use MacGyver and Neo4j to do that.
RVB: 07:33 Super cool. So, that brings me to the question that I ask everyone on this podcast, why Neo4j? Why a graph database to do what you're doing? Was there any specific reason for that or is there anything you want to call out that you really like about it in your current environment?
AS: 07:54 Definitely. I guess, first off, the low latency and it's really the ad-hoc querying is super, super useful. I think another thing that really stands out to me is how flexible and scalable Neo4j is. So, we started small just with app instances, but it's really easy to build new layers and new relationships on top of already existing ones, and so where we started with just app instances, now it's become this huge infrastructure mappings of so many different types of nodes. It's really cool how with Neo4j, your data set can really easily evolve and grow in terms of complexity or structure. It's just so easy to use and that's why we've been able to keep using it. And also, obviously, it's just really good at graphing relationships between things and mappings. That's where I really--
RVB: 08:52 Yeah, it makes total sense. Do you guys use Cypher at all? Do you do interactive querying or maybe--?
AS: 08:58 Yeah.
RVB: 08:57 Yeah, you do.
AS: 08:59 Actually, within MacGyver, we have a web interface for Neo4j and people enter and Cypher queries to look up stuff.
RVB: 09:06 Super cool. Very good. So, where is it going, Ashley? Where are you guys going to take Neo4j in the future? Any perspectives on that? I'd love to know more about that.
AS: 09:17 One unit's already a really integral part of my MacGyver and we're just using to hold everything together and-- MacGyver also has become a central point of information. As I said, as we move into AWS, we're going to keep putting all that stuff into Neo4j. We also are using it to track our asset management, and even as I was saying before, the infrastructure mapping. We could add network and database components into that and get-- just keep building the infrastructure map, keep building our AWS map. Another thing that's on the road map is to utilize those app instance check-ins to create a service registry for all of our apps. That way, we can keep track of who owns this app, or maybe if we map it to get [?] what Repo is it? What Jenkins job does this correspond to? Is this out public? So, we'll have a service registry of all our apps, where people can go and just find out info that otherwise would be difficult to pin down.
RVB: 10:22 Super cool. I think it's a great use case for Neo4j, and I'm so happy that you guys found your way to it and are getting good use out of Neo4j. So, it's really--
AS: 10:34 Yeah, me too.
RVB: 10:34 Yeah, it's really great and I think we'll wrap up the podcast for now. I really want to thank you for coming online and talking to us about it.
AS: 10:45 Of course.
RVB: 10:45 If you don't mind, I'll share a couple of links to your video from GraphConnect--
AS: 10:51 Yeah, of course.
RVB: 10:51 --on the blog post as well. And I look forward to hearing more about you guys as you guys expand, and as you guys grow the service with Neo4j.
AS: 11:04 Awesome. Thanks, Rik. Thanks for having me.
RVB: 11:06 Thank you so much, Ashley. Bye.
AS: 11:07 Bye.
Subscribing to the podcast is easy: just add the rss feed or add us in iTunes! Hope you'll enjoy it!
All the best
Rik
No comments:
Post a Comment