Why digital transformation requires real-time, event-driven applications and how to build, deploy, and manage them. Delivered by Blaine Mathieu, VANTIQ Chief Marketing and Product Officer, at the London IoT Expo on April 18, 2018.
Good morning everyone. My name is Blaine Mathieu. I’m with VANTIQ, an application platform company to help you build real-time applications for your enterprise. And we’re here to talk about event-driven applications that involve IoT.
But before we do that, maybe to step back a bit. Although this is a conference for IoT, AI, block chain. I don’t think most of you are here to build AI solutions or block chain solutions or IoT solutions. What you’re probably here for is one of two reasons: you’re either from a startup and your goal is to disrupt existing business models, enterprises to try to reinvent how the world works reinvented digitally, or you are one of those enterprises, and you’re involved in the so-called digital transformation initiative.
You’re trying to figure out how to keep from being disrupted by the digital natives, by the startups. You want to figure out how to achieve growth levels like Amazon and Uber and Facebook and Netflix. And you want to use these technologies, AI, block chain, IoT, event driven applications, all of that, as the foundation to help take your business to the next level.
And if you don’t do that, the writing is on the wall. The trend about large enterprises being disrupted dropping off the Fortune 500 list is inexorable, it’s accelerating. And we all have to do something about that.
Now, the key challenge, or one of the key challenges, in doing that I would summarize as being about complexity.
First, there’s software everywhere. We heard Marc Andreessen say a while ago, “software is eating the world”, and now that software is distributed in the cloud, still on premise, and is increasingly on the edge. “The Edge is Eating the Cloud” our friends at Gartner say, and that’s a challenge to build applications and solutions that account for that.
Of course, IoT is driving a massive increase in the volume of data that’s flowing into and around and between organizations. And what do you do with that data in real time to transform your business? And then most of your enterprises are still, despite advances in AI and machine learning, filled with people. And one of your fundamental jobs is to make those people more efficient, more effective while the machines become more intelligent to truly enable what I call human-machine collaboration, not just a human machine interface, but a collaboration between systems, software machines, and people. And that will be critical to transforming your businesses.
And then all the technology that’s floating around and it’s well represented in this room, blockchain being the latest, exciting technology, that we’re all talking about and thinking about. Natural language processing, augmented reality is another one. [We’ll] probably have an AR section to this show next year.
So, huge trends. And finally, your enterprises are full of legacy technologies, legacy applications, some that you’ve built, some that you’ve bought from other companies, or even if they’re based in the cloud, I would still consider them legacy tech.
So what do you do about that?
If you’re a CIO or a technologist, you’re the brain trying to make sense of all this, and/or you need something, you need a solution that can take all of these trends, all of this data, everything that’s happening in real time and sense, analyze it in real time, but it’s not just about real time analytics and nice dashboards flashing. It’s about taking an action in real time while the data continues to flow in and around your company. That’s a big challenge. And unfortunately, those applications of today, the legacy applications I was talking about, aren’t up to that challenge.
So-called request/response apps where a user or maybe another system makes that request, the request goes into the application, a database is searched or queried, a response returns. That’s fine for non-real-time applications like CRM systems and ERP systems that are fundamentally about getting reports out of a database. But those applications aren’t real time. They aren’t fundamentally agile. None of these will digitally transform your business. And you know it. They’re important. You need them in your enterprise, but they’re not going to transform it. So that’s not the solution.
So what is the solution?
Well, it’s so-called real-time event-driven application. So what is that? Think of it this way: you’ve got all that, as I said, the data flowing in from IoT devices, sensors. Also, from enterprise systems, you know, an ERP system puts off a PO that has been approved. That’s an event. A location of a car in your fleet moves from one city to another city. That’s an event. All of these data flows are events that happen in and around your business or organization that you can track and take action on. You can see some examples of events here and then you do something in real time called complex event processing.
So, you’re continually analyzing, crunching, doing something with those events, trying to find events of interest. Maybe where two or three events come together in a particular way means I’ve got to do something. And usually, you add external data sources into the mix as well: rules, constraints, maybe if you’re doing a real time fleet management application. You’ve got a real-time weather feed coming in that’s also being processed by the application. And then fundamentally, it’s about taking action. What are you going to do based on all of these flows of data coming in from your machines, from your devices from your people? You’re going to relocate somebody. You’re going to assign a technician to do something, and then maybe you’re going to work with that technician while she’s fixing the machine in real time and the data continues to flow in. That’s real time, complex event processing and most transformative digital business applications are built around this premise, built around this architecture.
That’s how Uber disrupted the taxi industry. It turned the taxi industry, the concept of transportation, into a real-time business where it continuously understands the location and status of the driver, the location and status of the passenger, and everything in between, all through the entire process from beginning to end. A great example. So, the bad news is, you’re probably imagining, if you are a technologist or a CIO, that’s great. But those are very hard to build. Building so-called real-time reactive applications is a challenge. You’ve got all these events coming in asynchronously, not in any particular order. And the data is streaming in.
First, we had databases then we had data warehouses. Now we have data lakes, God help us. You can’t take real time action in a data lake. It’s good for later analysis, and I’m not saying analytics aren’t important, they are, but if you’re trying to make a process happen now in real time, that’s not the solution. As time passes, the value of data decreases. We all know that. Your systems also have to account for the inherent and increasingly distributed nature of the solutions we’re building today. The edge is huge. You know, most computing over the next 10 years is going to move to the edge where issues like latency, connectivity, and being able to take an action in real time are far better addressed than they are in the cloud. It’s hard to build distributed applications on the edge.
And then finally, all of these systems we’re talking about, all of these use cases, involve people, and people are the most asynchronous, unpredictable sources of events around a business that you could imagine. It’s very hard to create software which can account for unpredictable people, for us. Companies have been trying.
This is actually the real example of a real-time global supply chain application built by a company you all know, I can’t say the name. And like many companies that have tried to understand and take action on events happening around their business, in this case, it’s managing their global supply chain, they choose a platform. In this case it was IBM, but it could be Azure, or AWS, it doesn’t really matter. And then within that platform, they’ve got a bunch of different tools and technologies, Kafka and Cassandra, and all of these are very complex technologies, for those that are familiar with them. And they’re continually being revised, some are open source, some are proprietary license applications.
Basically, to make this run, you need a team of gurus, experts in every one of these core technologies. Now, if you’re this company, you have those gurus. So, you can build this real-time application. Now the problem, though, is even if you’ve got the Gurus, you no longer have the time. You don’t have two years to build this real-time application, which is what this took. You’ve got two months, two weeks before another startup from over there is disrupting your business. So, even if you’ve got the expertise, you no longer have the time. So, we’ll get back to the solution in a minute.
Let me build out a sort of a standard use case again that we’re talking about. You have this notion of the events process. In many cases, the initial processing happens on the edge, not in all cases, depends on the scenario. But increasingly, it’s happening on the edge, on edge nodes, gateways, other devices that are sitting at regional locations, or close to where that data is generated. Then, you still have a master application, maybe it’s on prem, behind your firewall, in your private cloud, or in the public cloud. As I said before, we’re adding external data sources, supplementing the data feeds coming from the devices and other sources, and then finally taking an action. Of course, those actions can be machine to machine.
The average oilrig now puts out 4 terabytes of data a day, four terabytes a day from one oilrig [from] the sensors on that rig. And when the pressure and temperature are too high and the shake sensor starts to detect something’s happening, that machine better shut off all by itself. And that’s perfectly fine, but there are many scenarios where, if it’s not that bad, you want the person to be able to work in real time with the machine and address the problem without shutting it off. And while the machine is being fixed, that’s another data flow back into the system. This looks like a one-way process, but it’s not. It’s a cycle. It continues to flow back and forth.
So, there are many use cases for event-driven technologies. I talked about real time supply chain and that complex example earlier. Fleet management I mentioned, field service is another one, which is very obvious because it involves people and machines, again, collaborating together. These aren’t all IoT use cases though, to be clear. We have clients in financial services whose main use case is about taking these data feeds coming from a plethora of enterprise systems, enterprise applications that have been disconnected and they’ve built an event driven application to orchestrate the actions based on that data in real time. Taking all these data flows, and it’s, again, much more of an application integration. This isn’t a typical or just a middleware solution. This is about building applications that allow you to actually take actions on the data flows, whether they be from IoT sources or any other source.
Our friends at Gartner have been doing a lot of writing about event driven applications recently. It’s a very hot concept in technology. And as you can see from the quote here, Gartner fundamentally believes that if you’re a CIO, you’re trying to digitally transform your business. You should be thinking about event-driven applications and event-based architectures. So, a real time event driven platform has four components that you should be looking for.
One is: are you able to build mission critical applications with it? Does it support increasingly distributed architectures, cloud, on prem, and on the edge? Does it enable effective and powerful human-machine collaboration? And can you build these applications quickly and easily? So let’s dive quickly into each of those. First of all, at its core, most digitally transformative applications, as I’ve said, do complex event processing. They’ve got a real time, so-called, reactive event-driven architecture at their core. And in fact, Gartner says that by 2020, which is only two years away, half of all new enterprise applications, anything that’s user facing, will have to be written or rewritten as an event driven application using an event-based architecture. By the way, some of these reports are on the VANTIQ website if you want to download them.
I Talked about mission critical. So, if you think back to those use cases we had that I was talking about a couple of minutes ago, if you’re running your 24/7 real time global supply chain with an event of an application, you can’t take it down for six hours while you do a software update or while you’re updating one of the core technologies. There’s been a new release by your open source partner. That thing has to be running every day, all the time, 24/7. You have to build, update, enhance, and improve it in real time without taking it down. Of course, it has to be scalable and everything else, truly mission critical. And again, Gartner feels that it’s event driven architectures, applications built on those type of architectures that will truly enable the kind of agility, extensibility, and web scale that we require to digitally transform ourselves.
We Talked about distributed applications already. So, perhaps you’re building you know increasingly building the applications in the cloud. But then after you’ve built the application, you have to get it out to where it needs to run. Get it, in many cases, as close to the edge as possible. And you know Gartner, again, had a really interesting analysis recently titled “The Edge is Eating the Cloud”. And while very few of your organizations are actually doing edge computing right now, it’s still sort of a science experiment, it’s moving very rapidly, very rapidly into the real world. Edge computing is where IoT was a couple of years ago. I’ve been going to these conferences for years, and even as recently as a couple of years ago, most of what we were doing here were science experiments. They were POCs. They were in the lab. And now a lot of IoT is in the real world. And that’s where edge computing is right now as well in my opinion.
Now we’ve been talking about human machine collaboration: how do you make people, systems, machines, not only virtual machines like software systems, but physical machines like robotics and other things work together? Absolutely critical.
Great report by Accenture, “Reworking the Revolution”, check out this report if you haven’t already seen. [It’s] a very lengthy, detailed analysis of how enterprises need to be thinking how to make their people more efficient and effective by collaborating with increasingly intelligent machines. And if you do that, massive benefits to your organization. Accenture “Reworking the Revolution” check it out.
And Then finally, all of this is meaningless if it takes a massive team of gurus and two years to build these applications. Your company is being disrupted, or let’s be positive, you need to transform your company now. Right? The speed of business is now the speed of real time and so you need what the analysts are calling a high productivity, low code, no code, but not just to build regular applications with no code or low code, you need to be able to build event driven applications like we’ve been talking about visually, graphically with not a lot of code. And the analysts are now starting to write about a new type platform that enables that to happen for the first time.
It’s actually an important point because some of you that are maybe my age or are a bit older know about case tools and other visual modeling tools have been around for decades actually where you can design applications by sort of flow charting things on a white board. On an application, it might look something like that on the right. The problem is, once you were done, you generated the code for the application and then that was it. You never went back to the diagram again. That was done. You now had a bunch of code to work with, low level code.
The new generation of these high-productivity applications, you do all the building, all the maintaining the enhancing, the updating, always in this mode, so very different than the old concept of visual modeling enterprise applications that has been around for a while. And the difference, as I’ve been saying, can be startling.
So, here’s our example, again, from this global supply chain application that I was talking about months, or in fact, in this case, years to get to market. High complexity. You need a team of gurus to make this happen, or a very costly systems integrator. And no matter who you use, it’s going to take a long time. The alternative these days, using some of these new platforms and technologies, is radically different.
The difference is nontrivial. It’s a dramatic improvement. We sometimes say it’s a 10x improvement. I don’t think that’s true. I think it’s actually much more than 10x because it’s not only about building the application the first time, it’s about deploying it, again, in many cases, complex deployments on prem, in the cloud, and on the edge, sometimes all at the same time. It’s about maintaining and updating and improving that application continuously over time. So it’s not just the productivity of building it once, it’s about what it takes to have a total cost of ownership and total ROI over the lifetime of your business. So, I’m Blaine from VANTIQ. We enable the real time enterprise.
We’re at Booth 340, just around the corner, if you would like to see some of this in the real world. And it looks like we do have a few questions if you’d like to ask anything about real-time, event-driven applications or anything else.