Happy Hoppaversary!

Today, November 6th 2024, it’s been a full year (!) that I have been working for Hopsworks. As with any youthful startups, the road to world domination is a bumpy one, and the journey with our team has been no different, so far. But one thing’s for sure: after one lap around the sun, my interest in and enthusiasm for the work that our team produces on a daily basis, is as radiant as ever. In this article, I would like to explore what I personally find so interesting about Hopsworks, and why I find the company and its products so interesting and motivating.

Hopsworks’ underlying tech is super solid

At the lowest level Hopsworks has unique technology that originated in years of research: HopsFS and RonDB. At some level, the origin story of Hopsworks is actually firmly tied to the bringing together of these two technologies in one consolidated data infrastructure platform. This project is grounded in years of practical experience, and the basic realization that these two components enable some really interesting capabilities in Machine Learning and AI. Bringing these components together, and integrating them into the coherent architecture that is the Hopsworks Feature Store in its essence, is the secret sauce of what Hopsworks does. It’s super unique because it can be deployed anywhere, and boasts crazy good performance stats - if only because the underlying storage components (HopsFS and RonDB) have been optimized for the feature store workload.

Hopsworks takes a broader view

Ok - so the tech is great. But why does that matter? Well, to me, it seems like the “Feature Store story” is “just” the start. A feature store is more than just a technology component: it is an enabler for building better AI and ML systems, by applying all the lessons learned from DevOps, Agile software development and FTI pipeline architectures. Today, Hopsworks is an MLOps platform, that brings the people and the processes that we want to apply to professional AI and ML systems, together around a feature store. This is not trivial, because MLOps is about more than just the tech: bringing people and processes together is hard, as anyone who has ever worked on a complex project will know. We have found that the feature store can be a fantastic forcing function for MLOps: the data foundation will lead the way, and it will bring the people and processes together.

Hopsworks’ unique value proposition, on technical AND non-technical levels

Sometimes people think that an infrastructure product like Hopsworks will only be used by very technical ML engineers, data engineers or data scientists, and that they are the only persona that stand to benefit from this kind of implementation. This is a complicated message, because of course it is true and not true at the same time.

It’s true because data engineers and data scientists stand to gain a massive amount of productivity and professional satisfaction, because the Hopsworks infrastructure will simplify their infrastructure related tasks. Research has shown that technical engineers spend 30-40% of their time doing non-productive, infrastructure-related tasks. That number needs to come down if we want to have any kind of productivity in building these systems, and that is Hopsworks’ objective. By providing a unified platform for AI and ML, technical stakeholders can make their lives easier and more productive.

But it’s also not true, because Hopsworks serves two other, important stakeholders, and it does so quite significantly.

First, we serve the technical team managers, IT managers, project managers and budget holders to much more efficiently allocate their budgets. We do that by reusing artifacts (features, pipelines, models), but also by offering deployment flexibility (on-prem and in the cloud) that allows you to choose the right platform for the right workload. Cost savings of 100+% are not unrealistic there, on an annual basis. That is NOT small change!

And secondly, we facilitate ML and AI governance at a very profound level. By managing the data that is used in models and tracking all the different manipulations that are run on it as we prepare, create and deploy our models, we can work towards the required explainability and FAIR principles that our regulators are going to require, in all kinds of industries. Before too long, any organization, public or private, that wants to use AI/ML for business purposes, will need to demonstrate proper governance - and the MLOps infrastructure around the feature store that Hopsworks provides will be super useful for this. As such, it will enable EU AI Act, or any other regulatory initiative out there, compliance for our customers.

So let there be bread cake

These are the main reasons why I think Hopsworks is just the best thing since sliced bread, and why it has been a fantastic personal and professional challenge to work with this team for the past year. It’s not always easy to get your head around complex platform products like this, but after this year in the trenches, I feel like I have seen a lot, learned a lot, and that we are supremely well positioned to provide amazing value for our clients. Onwards, and upwards!