Uptake of Complex Event Processing (CEP)
I’m seeing more and more articles about complex event processing (CEP) these days. If you’ve followed by blog, you’ll know that I’m a big fan of events, so I try to read these when they come across my news reader. One of the challenges I see, however, is that event-driven thinking is not necessarily the norm for businesses. Yes, insurance companies may deal with disaster events, and financial services companies may deal with “life” events like weddings, births, kids going to college, but largely, the view is very service-based. It is reactionary in nature. You ask me for something, and I give it to you.
This poses a challenge for event processing to gain mindshare. While event processing is certainly the norm in user interface processing and embedded systems, it’s not in your typical business IT. Ask yourself- if you were to install a CEP system in your enterprise today, what events would it see?
The starting point that I see for events should merely be publication. Forget about doing anything but collecting statistics at the beginning. Since events don’t align with how we’re normally thinking, perhaps we should let them show us how we should be thinking. This gets into the domain of business intelligence. The beauty of events, however, is that they can make the intent explicit, rather than implicit. If I’m only performing analysis based on database changes, am I seeing the right thing, or am I only seeing symptoms of the event? Not all events may result in a database change, and that’s where the important correlations may lie. If some companies shows up on page one of the Wall Street Journal, it could result in increased trading activity for that company. My databases may record the increased trading, I may not have a record of the triggering event- the news story.
Humans are very good an inferring relationships between events, sometimes better than we think. But without any events, how can we infer any relationships? We don’t want to overwhelm the network with XML messages that no one ever looks at, but we shouldn’t be at the opposite extreme either. Starting with new applications, I’d make sure that those systems are publishing some key events based upon their functionality. Now, we can start doing some analysis and looking for correlations. We then start to encourage event-driven thinking about the business, and as a result, have now created the potential for CEP systems to be used appropriately.
As an example of how far we still have to go, let’s look at Amazon. They certainly leverage business intelligence extremely well. Unfortunately, as far as I can tell, it’s largely based upon tracking the event that impacts their bottom line directly- purchasing. If I were them, I’d be looking at wish list activity much more strongly. Interestingly, my wife gets more recommendations on technical books that I do. Why? Because she’s purchasing them as gifts for me. I put them on my wish list, she buys them. Because they’re looking at the wrong event, they now make an inference that she’s interested in them when she isn’t. I am. They need to track the event of me adding it to my wish list, along with someone purchasing it for me, and then turn around and make recommendations back to me. Of course, that’s part of the challenge, though. There is simply a ton of information that you could collect, and if you collect the wrong stuff, you can waste a lot of time. Start with a small set of information that you know is important to your business and build out from there.
11 Responses to “Uptake of Complex Event Processing (CEP)”
Leave a Reply
An interesting post.
But it may underestimate the role – and thereby the potential value – of event processing by implying that IT applications (or perhaps the interaction of services in a SOA?) are the only or primary source of events. If you accept that, then the failure to generate the events within IT may reasonably be seen as a throttle to the adoption of event processing technology.
However, one of the important use cases for event driven architectures is the ability to respond to events that are generated from outside the IT architecture (market data events from a stock exchange, RFID events, sensor datato identify patterns of significance within those events (when perhaps all you reasonably do is infer the significance) and then act – with that action perhaps being a new “event” that is a signal to the IT architecture to do something.
Thus, the adoption of event driven architectures may be less throttled by the need for IT to be more attuned to publishing of events in their architectures as it is for those architectures to “listen” more attentively to external events and respond.
Chris-
Great comments. Just as we must deal with service consumers and providers that are both inside and outside the firewall, we also need to deal with event sources and sinks that are both inside and outside the firewall. The question is whether or not a standards-based publication (not syndication) mechanism exists and is in use? Certainly, companies deal with market data feeds and other real time information sources today. Much of this is still proprietary, and not really in a standards-based envelope that could be given to a CEP engine. The semantic consistency is probably even greater of an issue. I don’t mean to sound pessimistic, because I’m not. Things are headed in the right direction, I just think it’s going to be an even longer uptake than proper utilization of SOA.
Thanks for your follow up. I think I would respond to your points in two ways:
1. There are indeed standardization efforts in process in many of the markets that are candidates for CEP. In financial services, initiatives like FIX and others are facilitating this greatly. RFID has EPCglobal activities. I am sure that other markets have theirs as well, perhaps facilitated by XML or other approaches. These efforts can help facilitate the effort to consume events by providing more standardized formats for events, as well as greater standards with respect to connectivity. These are all driven by the interoperability needs specific to those markets.
2. Fundamentally, the implementation of CEP and event driven architectures are driven by business goals. Given that, expecting some form of “lingua franca” for events is probably not realistic. A market event in finance is obviously differently constructed than an RFID event. Within financial markets, an equities event is different from a foreign exchange event, or one for derivatives or fixed income. I would argue that some of these differences are legacies that can adopt to standardization, but some are reflect the unique aspects of the industries. Given that, the onus is on the CEP platform to accommodate those differences, since those differences address the specific needs of the markets served by the events. It is likely that achieving semantic consistency – for purposes of processing the events in a CEP engine, would emasculate the value of the events for the particular markets. Thus, (full disclosure, I work for a CEP provider) I believe the imperative is for the CEP products to consume the events and, if needed, normalize them internally for purposes of CEP processing, not expect them to be delivered in a standardized format.
[…] Let The Users Use Your Events In his Outside the Box blog, Todd Biske writes about the (lack of) Uptake of Complex Event Processing. He points out several reasons for the lack of uptake — the primary reason being a shift in mindset that has not taken place in most businesses. […]
[…] The point I really want to make however, which expounds on my previous post, is that I simply think event-oriented thinking is the exception, rather than the norm for most businesses. I’m not speaking about events in the technical sense, but rather, in the business sense. What businesses are truly event driven, requiring rapid response to change? Certainly, the airlines do, as evidenced by JetBlue’s recent difficulties. There are some financial trading sectors that must operate in real-time, as well. What about your average retail-focused company, however? Retail thinking seems to be all about service-based thinking. While you may do some cold calls, largely, sales happen when someone walks into the store, goes to the website, or calls on the phone. It’s a service-based approach. They ask, you sell. What are the events that should be monitored that would trigger a change in the business? For companies that are not inherently event-driven, the appropriate use of events are for collecting information and spotting trends. Online shopping can be far more informative for a company than brick-and-mortar shopping because you’ve got the clickstream trail. Even if I don’t buy something, the company knows what I entered in the search box, and what products I looked at. If I walk into Home Depot and don’t purchase anything, is there any record of why I came into the store that day? […]
Todd – since you’re taking a strong position that event driven architectures will be the “exceptions” rather than the norm, could you give any sense to quantify your position? Hard to understand what “slow” or “rapid” adoption means to you? Do you think 1% of applications will be event driven? 10%? 20%? 50%? And since event driven architecture and complex event processing is quite new, what kind of time duration do you think is appropriate? What would constitute “rapid” adoption?
Mark-
Thanks for your questions. There are a couple of subtleties that I wanted to call out. First, there is a difference between event-driven business thinking and event-driven applications. The IT guys (and I’m one of them) can build systems on a pub/sub architecture today and the business will never know it. Second, in my other post, I did use the exception/norm phrasing, but again, that was in reference to event-driven businesses, not necessarily event-driven architectures. It also refered to the current state. That being said, my opinion is that both event driven architectures and event driven businesses are the exception today, not the norm.
As for the pace of adoption, I’d use SOA adoption as a reference. I started looking at SOA in 2003 and had thought that a journey to SOA would be at least a 5 year effort. It turns out that even that was probably optimistic. There were applications using services within a year, but there’s still a long way to go. Part of it is simply the way IT works. If there happens to be a big initiative that touches many systems, you may get lucky. If you don’t, it’s going to be a slow, gradual journey.
Now, if I look at EDA, there are more hurdles to overcome. I feel that SOA is a bit easier to relate to on the business side, and thus has greater potential for being broadly applicable. EDA, on the other hand, is going to be a bigger stretch for businesses that aren’t naturally event driven. So, in terms of mainstream adoption, I’d expect it to take even longer than SOA. How much longer? I don’t know. We’re just at the early stages, so it’s too difficult to say. There are certainly some areas where EDA is (or should be) mainstream, and thus there is potential for a marketplace today.
I hope this helps you understand my thinking. I don’t have “industry analyst” in my title, so this is strictly an opinion. I’m not going to offer up percentages, because that’s too easily shot down without analysis to back it up. Opinions, on the other hand, are a good way to open up the discussion. I’d like nothing more than to have this thread spawn some discussion on how some companies are leveraging EDA today for business advantage. As I’ve said, I’m a big fan, and will certainly be trying to encourage companies I work with to understand how they can leverage events where appropriate to improve their systems.
EDA is not a new concept. It has been around about as long as Service Orientated technologies. The standardization though is an interesting idea, and how it flushes out will be also interesting. I believe that it will eventually take hold simply because we love standards. It is more invasive than SOA, but only from a perception point of view. In many businesses today, events are actually more important than SOA. That was the original resistance to SOA because businesses did not run that way. This is good thing because now, companies can do business the way they are used to doing business, and simply apply a different architectural concept.
It should be defined pretty quickly because all the standards are already in place for publish/subscribe, event, complex event handling, etc. All these capabilities are supported today already in most of SOA vendors, and such it’s simply a matter of coming up with standards.
[…] Uptake of Complex Event Processing (CEP): This post from February 19, 2007 discusses my thoughts on the pace that major enterprises will take up CEP technologies and certainly raised some interesting debate from some CEP vendors. […]
[…] interesting, building on the natural relationship to SOA. As I’ve previously stated in my uptake of CEP post, I don’t think that most organizations are ready for CEP and EDA yet. The debates that […]
[…] of complex event processing (CEP) and event driven architecture (EDA), and discussed some recent comments from Todd Biske. Nice job Joe! Posted by progress apama in EDA and SOA , Followed with No Comments. […]