Everyone in the IT business knows about analytics and the value of using large amounts of historical data to make...
better business decisions. The challenge for some applications is that "historical" qualifier. Most analytics tools are designed to work on masses of collected data, not on data as it is being collected.
As the time between data collection, analysis and action shortens, analytics applications morph into something very different: complex event processing (CEP). The real-time intelligence provided by CEP applications can help an organization use data to modify current operations rather than just future ones. However, leveraging event-driven architecture effectively requires that users understand what CEP really means, the three basic models of CEP and their characteristics, and the intrinsic CEP limitations.
Key CEP terms
Let's start with some definitions. Events are things that happen in a business, including business activity, process status, network or IT conditions, or almost anything else. Many events can be handled as they are recognized, as would be the case with a fire alarm. CEP is used where individual events can't be processed reliably and must be analyzed in context. In nearly all cases, CEP analysis demands that events be associated in time and requires some form of accurate event time-stamping.
CEP applications tend to fall into two categories -- event correlation and root cause analysis. Typically, event correlation is used to recognize conditions that are signaled not by a single event, but by several events in combination, such as saying "If you sneeze and have a fever, you have a cold." Root cause analysis is used to interpret the underlying reason for a series of events -- usually failures -- to ensure remedial action doesn't chase symptoms rather than a problem.
All CEP is supposed to be real time or contemporaneous with events, but, of course, contemporaneous can mean many things. And the more complex the relationship CEP is working with, the harder it is to sustain a realistic claim of real time. There are three generally accepted models of CEP implementation, all of which balance sophistication, real-time operation and process costs.
Picking your CEP model
The simplest approach to CEP is trigger or threshold activation of processing. With this model, an event either directly induces some action, or events are counted to a threshold and action is taken when the threshold is reached. CEP can introduce event processing in an event flow from source to the normal destination, such as online transaction processing, because the delay generated is minimal. Although trigger or threshold CEP can be carried out on a single type of event, it's also possible to use several different events with different weights to provide some level of deeper understanding of conditions.
The second model is the contextual model, which presumes that an event or event set is meaningful in a specific context and thus maintains that context. This can be done through pattern-matching, meaning looking for a specific set of events that fit a pattern, or through state-event processing where events change an explicit context or state, and later events are interpreted in context. This latter approach is commonly used in network management, where context examples might include initializing, active or fault.
For even more complex CEP, it's possible to employ a cascade analysis model where event streams containing one or more types of events are preprocessed using one of the CEP models. They are not processed to take action directly, but rather to generate other events or event contexts that are then fed into another stage of CEP until eventually actions can be decided.
Tips for effective CEP
It's a mistake to underestimate the complexity of CEP needs when laying out an application because picking too simple a model will usually result in imperfect event analysis and increased difficulty in maintaining the application, especially as new conditions are recognized. Users report, for example, that about half of all CEP applications end up benefitting from cascade analysis, but fewer than 1 in 10 start out that way.
Implementing an event-driven architecture or CEP is more than just starting with the right model. Remember: You can't analyze events properly if they're not flowing through a common point, so event aggregation may be an essential step. Also remember that as the model of CEP evolves from simple thresholds to cascade analysis, the time required to analyze events and make a process decision will increase. The events to be handled become more complex, so the CEP application must be designed more carefully, and more processing power must be devoted to the analysis.
It's also important to segregate CEP applications based on whether CEP is the primary driver of application processing or whether events already targeted at other applications are being analyzed for additional process management. Security is a good example of the latter: You are securing something by intercepting events and looking for signs of intrusion or improper use. The primary application cannot be adversely impacted.
One thing that can help ease the introduction of CEP into established transaction or event flows is duplication or summarization of events. It may be possible to use a simple CEP model of threshold analysis to do primary event handling, and have that model generate a secondary set of events that will be dispatched outside the normal transaction and event flow to CEP processes. This will avoid duplicating an entire event flow, which could be expensive in terms of resources, and limit the impact of CEP on existing applications.
Remember that CEP is based on contemporaneous establishing of event contexts. Analytics is based on historical recovery of context. If you don't need to take an action in real time, then analytics is a less expensive approach and also one that can easily uncover unexpected conditions or situations. CEP is designed to respond to specific things and isn't suited for data mining models of analysis. However, used right, CEP complements analytics and moves some analytic processes closer to real time and brings a new dimension to IT support of business operations.
Understanding CEP and BAM
How to gain a competitive advantage with CEP
How to get data moving with CEP