Alignment (or not) between data tech and software development industries

How data technologies and software development have evolved in parallel and synergistic ways

When I think about both the data industry and the software development industry, and try to take stock of the evolution that we have gone through in them in the past 25 years, I can’t help to see that there are some really interesting and important trends, and that these trends are actually really well aligned between both industries. No surprise?

Relational databases and waterfall development dominating the earth

When I started my career in tech, the world was reigned by Relational Database Management Systems (RDBMSs). My university graduation thesis was actually about a sizing study of an early version of MS SQL Server – of all things J … And: at that time, we were developing software in a very well-structured, very well-governed fashion, using waterfall development methodologies. Every step had its clear and defined purpuse, and would follow another step. It was all about analysing, designing, developing and testing software on significant chunks of functionality, that would all take months to deliver.

Things started to shift…

I remember at the time, feeling that there was starting to be a shift. We were starting to see some understanding of how we would need to become more agile – that the mythical manmonth was indeed, mythical. That we would need to break up problems into smaller ones. That use cases were going to have to drive our alignment with business stakeholders.

In my experience, the advent of many new technologies (object orientation, open source operating systems and development frameworks/infrastructures, to name a few), this started to change significantly, and we started to develop software in a truly more agile, iterative way. And forgive me if I get the chronology wrong here, but just around that time, we also started to look at data in a different way, and our industry developed an entire range of new (NOSQL, for lack of a better category name) data technologies. All of a sudden the normalised, table-based data model that had famously overthrown the navigational and network databases of the mainframe, was deemed less opportune for modern software development practices. Document databases, key-value stores, graph databases, and wide-column stores each had their unique traits that suited specific use cases – and therefore drove their adoption to sky-high levels. We are still seeing that today.

Parallels between data tech and software development

So in my mind, we have a really interesting, and parallel set of shifts:

• From waterfall methodologies to agile software development:
  
  
• From relational databases to NOSQL databases
  
  
  

About alignment and misalignment

It seems to me that there is an interesting alignment and misalignment here between these shifts:

• waterfall/agile methodologies are opposed and therefore misaligned,
• relational/NOSQL technologies are very different and therefore misaligned,
• relational/waterfall were clearly historically aligned, and share some characteristics for sure,
• NOSQL/agile were also historically aligned, and share some characteristics too.

I tried to express this in the following bigger mindmap, from which I have pasted above:


Interesting, not? Clearly, my overview and articulation of this evolution is incomplete and even inexact, but the broad lines of the argument hold true. I would love to hear it if you think otherwise. Note that I do not mean to say that you cannot do agile with RDBMSs, or that you cannot do waterfall with a NOSQL database – but I do not think they are well aligned.

It feels important to point out these types of higher level shifts so that people understand why a modeling tool like Hackolade, does what it does, and doesn’t do what it doesn’t do. After all: data modeling

• On the one hand touches the software development industry: software developers will use data models to better understand their business domains, and develop better software in an agile, iterative way. Hackolade is super aligned with that approach.
• On the other hand touches the data industry: data architectures are more and more frequently composed out of different, heterogenous database systems, streaming services, APIs, and many other components. Hence Hackolade's support for all kinds of different physical targets.

And in any case: let’s hope that it encourages the conversation and discussion – we can only collectively get better by doing that.

All the best

Rik

PS: I thought about using a graph method and "close the triangles". Kind of like the old adage "an enemy of my friend is my enemy", one could argue that "misalignment of a concept to an aligned concept will misalign that concept with the original concept". Therefore: NOSQL/Waterfall and RDBMS/Agile would be implied to be misaligned. But I think that is too simplistic, and therefore I did consciously not include it in the article.