The myth of the 95% AI failure rate

 

Why your AI project is actually failing–and how to fix the foundation

You’ve probably seen the headline by now: “95% of generative AI pilots fail to deliver measurable ROI.” It’s been included in just about every article and presentation on the AI topic recently - and it’s most definitely sparked a familiar fear – that we’re living through another overhyped AI bubble.

But that number, taken at face value, misses the real story. Because what’s failing isn’t AI. It’s how organizations are trying to adopt it. Once you look past the headline and into what the research actually measured, a different picture emerges—one that’s far more practical, and far more fixable.

Here is a critical breakdown of what the research really showed, what it did not show, and the non-negotiable architectural mandate required to transition your projects from the volatile 95% bracket into the successful 5%.

1. What the research really showed: A failure of integration, not intelligence

The MIT Media Lab’s Project NANDA (who advocate for decentralised web of AI agents (see their projected evolution in the chart below), and as such are somewhat biased towards the current state of AI) report defined “failure” precisely: the inability of the AI pilot to transition beyond the proof-of-concept stage and achieve rapid revenue acceleration or a substantial, measurable return on investment (ROI). 

In other words, these weren’t models that didn’t work. They worked just fine in controlled environments. However, they failed when they hit the real world.

The researchers describe this as a “learning gap” – the moment when an AI system leaves the lab and runs into fragmented data, unclear ownership, and workflows that were never designed for intelligence to plug into them.

So the takeaway isn’t “AI doesn’t work.”

It’s “we’re dropping AI into environments that aren’t ready for it.”

Why AI projects stumble: it’s usually not the tech

When AI initiatives stall, the real roadblocks are usually internal – specifically how we try to fit the technology into our existing organization and where we decide to spend the money.

First, let's talk about the Integration problem. 

Dropping a powerful but generic tool like a Large Language Model (LLM) into a complex company is like trying to use a foreign body in a human system – it just doesn't mesh. For these tools to actually work, we need more than just the software; we need to break down the departmental silos, establish clear governance, and define exactly how the new AI connects with our current systems. Without that framework, the tech is essentially an outsider that can't access the necessary context to be effective.

The second big issue is the strategic misalignment, which I call the "Visibility Trap”, which also indicates a strategic mismatch in where money gets allocated. 

A large share of AI budgets flows toward visible functions like Sales and Marketing. But MIT’s own data shows the highest measurable ROI comes from back-office automation – reducing repetitive internal work and operational drag The conclusion is clear and simple: we often fund what looks impressive instead of what actually compounds value.

Bottom line: the AI isn’t failing. The organization is failing to prepare the ground it’s meant to operate on.

2. The core obstacle: the data readiness crisis

If strategic failure kills ROI in the pilot phase, data fragmentation kills the rollout in the production phase. Gartner data supports this operational challenge, showing that only 48% of AI projects successfully transition into production.6

The single biggest blocker to rolling out AI is the state of enterprise data, which we can dissect into three components:

1. The unstructured data problem:
   Most enterprise data – emails, tickets, documents, logs – is unstructured. It lacks consistent context and labeling, making it unusable for reliable, auditable AI unless it’s heavily cleaned first.

2. The fragmentation trap:
   Customer context is spread across CRMs, ticketing systems, engineering tools, and ERPs. Stitching this together with brittle API calls doesn’t scale. It slows systems down and introduces failure points.

3. The trust gap:
   When leaders can’t trace answers back to source data –or predict how the system will behave – they won’t rely on it for real decisions. That’s when projects quietly get shelved.

Without good data, no good decisions. That eternal truth is why so many promising projects end up getting abandoned - the lack of good data makes it so that they are simply set up to fail.

What do the successful 5% do differently?

To succeed where 95% of companies stall, you must pivot from model-centric optimization to a data-centric architectural foundation.7 This requires securing three critical capabilities: Context, Trust, and Action.

This is where a platform like Computer by DevRev fundamentally re-architects the problem, ensuring your AI initiatives are grounded in a ready-made, unified data layer.

How DevRev addresses the foundation problem

Let’s walk through the three things that DevRev does, as part of our yearlong effort to architect Computer from the ground up for this purpose, very differently.

1. Context: fixing fragmentation with a unified data layer

DevRev replaces fragile data federation with physical consolidation. Using bi-directional sync, data from CRM, support, engineering, and other systems is continuously pulled into a single store. That data is then structured as a relationship-rich knowledge graph – linking customers to tickets, tickets to code, and code to documentation.

This turns scattered, unstructured data into something AI can actually reason over.

2. Trust: grounding AI in auditable queries

To address governance and hallucination concerns, DevRev grounds conversational AI in an auditable layer. Natural-language questions are translated into standard SQL queries against the unified data layer. That means answers are:

• Predictable

• Traceable

• Verifiable against source data

This is what makes enterprise-grade trust possible.

3. Action: closing the learning gap

Insight alone doesn’t deliver ROI. Because DevRev is a system of record, its AI agents are write-enabled. They don’t just answer questions – they can update tickets, log bugs, assign ownership, and execute changes inside real workflows. That’s how the “learning gap” closes: insight turns into action, and action turns into measurable operational impact.

The real lesson behind the 95%

The 95% failure rate isn’t a warning about AI, really. It’s a warning about treating AI like a plug-in instead of a system. GenAI success depends on foundations – context your AI can understand, trust your leaders can audit, and actions that move work forward automatically. When those are in place, AI stops being experimental – and starts compounding value.

If you want to dig deeper into DevRev, Computer, or any of the ideas here, I’m happy to continue the conversation.

All the best

Rik

OpenTrials in Neo4j - with a simple ETL job

I have been meaning to write about this for such a long time. Ever since the lockdown happened, I have been wanting to take a look at a particular biomedical dataset that looks extremely interesting to me: the OpenTrials dataset. If you are not familiar with this yet, this is what they say:

OpenTrials is a collaboration between Open Knowledge International and Dr Ben Goldacre from the University of Oxford DataLab. It aims to locate, match, and share all publicly accessible data and documents, on all trials conducted, on all medicines and other treatments, globally. 

It's a super interesting initiative, and it really flows from the idea that in much of the very intensive, expensive biomedical research, we should be looking at how to better use and re-use the knowledge that we are building up. Kind of like what people in the CovidGraph.org initiative, het.io (remember the interview I did with Daniel - so great!) and others are doing. 

Downloading and restoring the dataset

It's a bit hidden, but you can actually download a (slightly older, but still) dataset of the OpenTrials dataset from their website. The dataset is actually a Postgres dump file: I got the latest one from http://datastore.opentrials.net/public/opentrials-api-2018-04-01.dump. 

Working with the ICIJ Medical Devices dataset in Neo4j

Just last weekend our friends at the ICIJ published another really interesting case of investigative journalism - tracking down and publishing the quite absurd and disturbing practices of the medical devices industry. The entire case with all of the developing stories can be found at https://medicaldevices.icij.org/ - take a look as it really is quite fascinating. Of course that meant that I wanted to see what that data looked like in Neo4j, and if I could have a play. I didn't have time for a full detailed exploration yet - but hopefully this will also give others the opportunity to chime in. So let's see.

The Medical Devices dataset as a graph

This turned out to be surprisingly easy. Just download the Zip file from the ICIJ website: https://medicaldevices.icij.org/download/icij-imddb-2018-11-25.zip, unzip this, and then we get 3 comma-separated-values files:

• one for the Devices that are being reported on
• one for the Events that are being reported (whenever something happens to a device (eg. a recall) then that is logged and reported)
• one for the Manufacturers of the medical devices.

That's easy enough.

Podcast Interview with Estelle Joubert, Dalhousie University

One of the coolest things about Neo4j is just the sheer breadth and diversity of applications that we see for connected data and graph databases out there. I think I have said it before, but it truly continues to baffle me. Very frequently, I will have a morning conversation with a user about battling financial fraud, a lunch conversation about using graphs in biotech to fight world hunger, and an afternoon conversation about real time recommender systems in retail. And of course finish it of with a beergraph conversation in the evening :) ...

Really - it's just amazing. And the next podcast episode is a true testimony to that. I got to have a chat with a lovely lady all the way over in Canada recently, Estelle Joubert from Dalhousie University. She and her team have been using Neo4j in her amazing field of research, which is all about understanding how music and opera came to be what they are today in a historical perspective. She is best at explaining it herself - so here's our chat:


Here's the transcript of our conversation:

RVB:  00:01:20.209 Hello, everyone. My name is Rik, Rik Van Bruggen from Neo4J, and tonight I am joined by a guest on our podcast all the way from Canada, someone that has been working with, and experimenting with, Neo4j for quite some time in a very interesting domain that I hadn't heard of before. And that's Estelle Joubert from Dalhousie University. Hi, Estelle.

Podcast Interview with Gábor Szárnyas, Budapest University of Technology and Economics

Waw. That was probably the longest stretch that I went without publishing blogposts or podcasts over here. I have no real excuse - the start of 2017 has just been super busy and interesting - with a lot of travel that does not really help with quiet "writing" time. But it's all great fun - I just need to get back into the rhythm - and today is the start of that.

Today's podcast is actually super cool. It started at a beautiful Brussels bar after Fosdem. At this conference, there have been "graph devrooms" hosted for the past couple of years - and this year it was a really nice lineup.  One of the speakers, Gábor, did this really interesting talk about "Graph Incremental Queries with OpenCypher", which is really cool. So after the conference, it turned out we share a passion for cycling too - and we decided to get together for a nice recording. Here it is:


Here's the transcript of our conversation:

RVB: 00:04.202 Hello everyone. My name is Rik, Rik Van Bruggen from Neo Technology and I must confess I feel very, very guilty now because this is the first time that I'll be recording a podcast in 2017, so happy new year. In spite of the fact that it's Valentine's Day. But yeah, I was slacking a little bit but I want to bring the podcast back to life and I've lined up a bunch of people to help me with that. And today I've invited someone who I've who only met like two weeks ago at the FOSDEM Conference in Brussels. And that's Gábor Szárnyas from Budapest. Hi Gábor. 

GS: 00:42.680 Hi Rik. Nice to be here. 

RVB: 00:43.500 Hey. Thank you for joining me. It was a great time meeting you in Brussels over some Brussels beer, but yeah we talked to each other about your work and I thought it would be great to have you on the podcast. So my first question is going to be who are you, and what do you do? What's your relationship to the wonderful world of graphs? 

GS: 01:10.158 Okay. So I'm a researcher at Budapest University of Technology and Economics. And also visiting researcher at McGill University in Canada. Now I'm working on finalizing my PhD, so hopefully I will be finish it within a year or a half. And I worked basically on graph- related topics in my PhD. 

RVB: 01:33.134 Oh, very cool. And don't forget you share another passion with me. 

GS: 01:38.380 Yeah, I'm also a cyclist. 

RVB: 01:40.152 Yes, exactly. 

GS: 01:40.729 So I started road cycling three years ago and it absolutely wondered me. I really like cycling-- 

RVB: 01:49.279 Same for me...

GS: 01:50.351 --and that's my main passion. 

RVB: 01:51.948 Same for me. We have a couple of other graphistas that are super passionate about cycling so we'll have to do a ride sometime. But tell us-- 

GS: 01:59.412 I agree. 

RVB: 01:59.558 --a little bit more about your work with graphs. What's it all about, what's your PhD about, and what are you working on? 

GS: 02:07.503 Okay. So my PhD revolves around three topics that are related to graphs. The first one is how to incrementally query graphs. So imagine that you have a complex query and you have a huge graph. Now obviously, it's very difficult to evaluate a query on the graph at a very short amount of time. So basically, as a workaround, we do incremental queries, which means that if your graph changes slightly then we maintain the result sets. And this is useful for a number of scenarios. You can use it for static analysis of code bases, you can use it for runtime modelling, you can use it for fraud detection, and so on. There are many use cases that present this scenario. 

GS: 02:52.025 The second topic of my PhD is how to benchmark an incremental graph query engine. Because, obviously, once you have an incremental graph query engine, you would like to have some feedback on its performance. And you would like to use that to continuously improve your query engine. So, with my research group, we designed and implemented a framework that allows users to do just that. Compare incremental graph query solutions to each other and to other competitors. 

GS: 03:22.765 And the third one-- yes? 

RVB: 03:22.870 Is that related to the LDBC work, the Linked Data Benchmarking Council, is that related to that? 

GS: 03:30.529 So basically they have similar goals. I was actually at Walldorf last week at LDBC Technical User Community Meeting. And LDBC has a couple of benchmarks, but currently none of those covers incremental graph queries and complex graph pattern matching. I talked to the LDBC guys and also attended the talks, and it seemed that there will be a new LDBC benchmark, which will have similar goal than my benchmark. And that will be called the Business Intelligence workload for the Social Network Benchmark. And the problem with that is that it's not yet ready. So I talked to it's core developer, Alex Averbuch, and he said that it will be ready within half a year but they are still heavily working on it. 

RVB: 04:29.082 Okay. But you had said that you had three goals, right? You had the incremental queries and then the benchmarking and what was the third one? 

GS: 04:34.976 The third one is closely related to network theories. A network theory is something that came up in the late '90s in the early nodes when people started to analyze graphs. So they took a graph of people where the nodes were the people in a community and the relationships were if they were friends or not. Or they took the graph of the World Wide Web where the nodes were the web pages and the relationships were the links between the web pages. So they took all these graphs and started to analyze them, and they derived very interesting properties, chief among which was the scale-free property of graphs. There are many papers on scale-free networks, and they discovered that this is very common in biology, in sociology, also in physics and other sciences. 

RVB: 05:28.488 What does that mean, scale-free networks? What does that mean?
GS: 05:30.744 So basically scale-free network means that the degree of distribution of the nodes follow the so-called power law. So you have very few central hubs. And basically, if you remove these hubs from the network then your network will break down to smaller components. And they discovered that this is how societies are organized, this is how citation networks work, and this is how power grids work as well. 

RVB: 06:00.783 Oh wow. Just like a universal structural characteristic of lots of networks. 

GS: 06:06.958 Yes, lots of networks. Obviously you cannot apply to all of the networks but it was a very big surprise to the scientists who worked on it that a lot of networks exhibited this property. So how does my PhD research relate to that? Well interestingly, there wasn't much work performed on tide graphs. So if you see Neo4j graphs, you obviously see that you don't only have people and websites and books, but you have all these inner single graphs. So you have tide graph, and they also have different relationships between them. And only in the last five to ten years have been there research about how to characterise these graphs. These have many interesting names. Some people call them the multiplex networks, others call them the multidimensional networks or multilayered networks. Analysing these is very tricky because obviously you have another dimension of complexity by having to deal with all the types of the nodes and the relationships in the graphs, but it's kind of a green area and you can do a lot of interesting work in it. I actually applied it to engineering models, so my research group works in model driven engineering. And there are engineering models for software, hardware, state machines, system design and so on. And basically we took all these models and analyzed them and we looked for some interesting properties. 

RVB: 07:58.123 Wow. 

GS: 07:59.168 We didn't find any huge results so we didn't find that these models are scale-free or they follow some very famous distribution. But we did have some interesting results on how to characterize these models. 

RVB: 08:18.190 Wow, very cool. So could you tell us a little bit more about how you got into the graph business, or the graph science if I may call it that way? How did you get into it, and why did you get into? 

GS: 08:35.661 Okay. Well, that's an interesting question. I think it started in 2011 when I had to pick my first individual research topic at my university, and my roommate
suggested that I should give a try to node secure databases. I was already very interested in anything that's related to databases, relational or not. So I started to work on node secure databases. And then I soon discovered Neo4j and the property graph data model. And I think what really struck me is how intuitive the graph data model is. There is actually a paper by Marko Rodriguez, who was the implementer of the TinkerPop framework, and he said that graphs are very intuitive because they describe the way that people use when thinking about the world. So people tend to abstract the world as things that are somehow connected. And you can perfectly describe this with graph nodes and graph relationships. So this is something I really like about graphs. And that's something that you also mentioned in this podcast, I think a couple of times, that you can use a whiteboard and then just start brainstorming, and having ideas, and drawing a graph. And you can use pretty much the same graph in your applications as well. So that's my favourite thing. 

RVB: 10:07.046 Jokingly, I always talk about my own acronym, which is WYDIWYS, what you draw is what you store. 

GS: 10:14.439 Yeah, that's a catchy acronym actually. 

RVB: 10:18.913 It's been repeated so many times on this podcast but it is a very big strength of graphs, right? The model is so intuitive and so descriptive, so rich, really. That makes a whole lot of difference, right? So I'm reading that that's also how you got into it, right? That's also why you think it's very valuable? Is that right? 

GS: 10:43.860 Yes. So basically after I got a bit familiar with the topic, I started my master's at university. And already during my master's I was working on the incremental query engine that I'm still working on today. So it's quite a long project. I've been doing this for five-plus years. And I really liked my experience during the master's so I joined the PhD and I just finished PhD school three weeks ago. So now it's only-- 

RVB: 11:11.500 Congratulations [laughter]. 

GS: 11:13.087 Thank you. So it's only up to me to publish some more papers and polish a dissertation. 

RVB: 11:21.283 So what does the future hold, Gabor? Where is it going for you personally? Where is your research taking you, but also how do you look at this taking ground in the broader industry? What's the future hold if you had a crystal ball? 

GS: 11:36.571 So, I would really like to be an academic. I really enjoy working at university because you have so many positive experiences with students. You can pretty much follow your own dreams and do research in almost whatever interests you the most. Obviously you have to fit within your grant proposals and your funding but this still gives you a lot of way to be creative and I would like to be a university lecturer and researcher in the future. So that's my kind of dream career. And-- yes? 

RVB: 12:17.317 And is it lecturing and teaching about graphs then or is it on a broader topic or is it computer science or what will be the topic then? Or topics? 

GS: 12:26.893 Well, I'm pretty much happy to teach anything relates to computer science, so I've taught topics from database theory to automata theory, system modelling, and software engineering topics, and also some laboratories on actual technologies. So our university is a bit of a mix between computer science and computer engineering. So we teach both theoretical and practical stuff and this is something that I also really enjoy. 

RVB: 13:01.647 Super. And what about the wonderful world of graphs and graph databases, is there anything like that in your future you think? 

GS: 13:10.251 Yes. So I really would like to get a version of my graph query engine that can be used by other researchers. I obviously understand that implementing production-grade software is not really possible within the limits of a PhD. But I would like to release a system that can be used at least by other researchers, both in academia and both in industry. I talked to a lot of people about this and it seemed that people would actually be interested in trying such a system, or benchmarking such system, and see how it works for their use cases. 

RVB: 13:49.818 Super. So final question, what's your favourite cycling destination? 

GS: 13:54.706 Ooh, that's a tricky question [laughter]. 

RVB: 13:56.737 Curveball for you. 

GS: 13:56.958 But actually, it's not a very common answer. I live next to the Hungarian-Austrian border, so I do go a lot to Austria because Austria has the best roads in Europe, and also most of the country is the Alps. So I live next to the lower Alps section, but even there you have very nice hills, and drivers are really polite, and you have these super flat tarmac all over the country. And that's what I really enjoy and I'm really looking forward to the summer. So I just usually disappear from the university for a couple of weeks and then go home and cycle. 

RVB: 14:38.375 Excellent. So no cobblestones for you? Unlike Flanders Classics or something like that? 

GS: 14:44.387 I actually really like riding the [inaudible], so I live in the inner historical district of Budapest and we still have a lot of cobblestone roads. And when I just started cycling in Budapest just to get to work and commute I usually tended to avoid those sections. But since I'm more into cycling I just go for the most cobblestoney sections [laughter]. This is something that you learn to enjoy or at least you think you enjoy it. 

RVB: 15:16.963 Yeah, yeah. Exactly. Very, very cool. All right. Well, I hope we get to ride one day together, that would be great. I really enjoyed this conversation. Thank you for taking the time. And I look forward to meeting you again someday, at FOSDEM or somewhere else. 

GS: 15:32.360 Thank you, for an invitation and we should definitely go for a ride. 

RVB: 15:36.138 Absolutely. Thank you, Gábor. 

GS: 15:38.717 Thanks. Bye

Subscribing to the podcast is easy: just add the rss feed or add us in iTunes! Hope you'll enjoy it!

All the best

Rik

Podcast Interview with Daniel Himmelstein, University of Pennsylvania

So today's podcast episode may well be one of the most interesting that I have ever published - and we have had some darn interesting episodes, if you ask me :) ... I got to know our guest, Daniel Himmelstein, by his great graphgist on "Drug repurposing by hetnet relationship prediction". Really interesting stuff - and Daniel actually got his PhD on this topic too. I found this video of his Thesis Seminar if you want more detail:

But for now we will just have a great conversation about his work. More interesting links below in the transcription - as usual.