In NOSQL, data modeling doesn't have to be logical!

At Hackolade, I have been really enjoying my journey into the world of data modeling. It really has been a journey: in some ways, I have felt like I had to go back in time a little bit, and re-learn some of the skills that I had known in the old days when relational databases still dominated this blue planet. ER diagrams, documentation requirements, naming conventions... all good things that had seemed completely normal in the old days - and that got a little bit of a bad rep when the hipster NOSQL databases and data formats started to gain popularity. I mean: at Neo4j, we used to say that "your data is your model", and downplay the need for deep modeling thoughts ahead of time. And at the same time, we also realised that every time a project wasn't going great, it was the freakin' data model that was the culprit. Every time.

The 3 concerns in data modeling, and their value

In a previous article, I have already alluded to the fact that I think that the confusion around the three words (model, schema, and metadata) is really important to clarify and get straight for our industry. That confusion has a historical reason: in the past 4 decades, when Relational DBMS’ ruled the earth, there was no real need to be specific around these three different concepts: the model and the schema were basically intertwined. But that has changed with NOSQL, and today we do really have to be more specific because we have more variability – we can have more or less modeling, and more or less schema, as we choose.  Tools like Hackolade and its polyglot data modeling have already made that very clear as well.

The main reason why I think being specific around this is important, is because it also allows us to be specific about the underlying concerns that speak to these words. 

Confusing words in Data Modeling

For a few weeks now, I have been helping out my friends at Hackolade with some really interesting work around their core product, the Hackolade Studio and how to make it even more successful in the marketplace. This means talking to a LOT of people - both current customers, active users, partners, friends in the industry - and more. It’s been fantastic to talk to so many interesting people, and to learn so much from their insights.

During these conversations, I have noticed that there’s quite a bit of work to be done to clarify and straighten out the meaning of the words that we use in these conversations. I have noticed that in the NOSQL data modeling space, we are not always very precise with our words - and that this imprecision can lead to all kinds of misunderstandings. Specifically, I have been struck by the confusion around 3 words: model, schema, and metadata.

The Agility Angst

First of all: let me start by showing my age. I was born in 1973, and yes, that does mean that there's a big, and I mean BIG, party coming later this year. Big Five Oh looming around the corner - and to be honest I am very, very fine with that. I am happy with my age - except when I am being dropped like a baby at the back of the cycling bunch. Then not so much - dammit.

What does my age have to do with anything? Well - it basically means that I grew up with a very, very different type of software development practice. OO was still young. The mythical man-month was still a thing. And methodologies all depended on strict, rigid waterfall development models.

Nothing wrong with the odd waterfall, of course, but with the emergence of more and more, and better and better, software tools, libraries and frameworks - this whole idea of the time consuming waterfall process has kind of become irrelevant. Who would still go through this entire process and ask their users to wait for months before they would be able to give any feedback? What users would accept that type of arrogance on behalf of their IT department? Seriously?

The modeling mismatch

After having spent 10+ years in the wonderful world of Neo4j, I have been reflecting a bit about what it was that really attracted me personally, and many - MANY - customers as well, to the graph. And I thought back to a really nice little #soundcloud playlist that I made back in the day: I basically went through dozens of #graphistania #podcasts that I had recorded, and specifically went back to the standard question that I was asking my interviewees on the podcast: WHY do you like graphs??? WHY, for god's sake!!!

Unsurprisingly, people very often came back with the same answer: it's the DATA MODEL. The intuitive, connected, associative, visual, understandable structure that we humans love interacting with: the labeled property graph (LPG). Have a listen to what people were saying:

Graphistania · The Best Thing about Neo4j