An attempt at demystifying CEP, BPM and BRMS

-- Posted by Carole-Ann

The more I read on Complex Event processing (CEP), the more I believe that people are confused or people are trying to confuse end users.  It reminds me of the Business Process Management (BPM) / Business Rules Management Systems (BRMS) market a few years ago: BPM vendors said that the rules could be modeled in the process flow and BRMS vendors said that processes could be implemented as ruleflows...  It took a few years to realize that each technology had a sweet spot and a real purpose.  They were complementary.

Now looking at the CEP Blog from Tim Bass, comments posted here (look at the comments) and there on the blogosphere, it is obvious that the role of CEP is unclear.  I am not disputing the value of the technology but merely begging the industry to keep things simple and straight-forward.

I will provide a simplistic prospective here.  You may argue that it is too simplistic but the more we discuss it, the stronger I feel about it.

Simple CEP / BPM / BRMS model

In our BPM / BRMS world, we came to the conclusion that we needed two types of technology: BRMS tools are used to make business decisions that are owned by the business users, BPM tools are used to carry out a process, executing on the decisions made by the BRMS tool.  Can you make a decision as part of a flow?  Yes.  Is it good practice to maintain them there?  No.  Can you automate a flow in a rules service?  Yes.  Is it good practice to do so?  No.  I assume we are all in agreement so far.

When CEP comes into the picture, we feel compelled to question this model.  Is CEP the right technology for processes?  Is it the right technology for decisions?  Some go as far as questioning whether CEP should replace Business Intelligence (BI) and/or Business Activity Monitoring (BAM).

At that point in time, being a product manager by trade, I get back to the problem definition.  what are we trying to accomplish?  Complex Event Processing...  This is all about processing events.  When is it valuable?  When you receive a flurry of events that cannot be handled efficiently using other traditional methods.  This is actually what happens most of the time in the real world: lots of things happen all the time.  Having a module that processes these events real-time would have tremendous value.  It would help dictate what kind of processes need to be executed and what types of decisions need to be made. 

Do we have to worry about an event overload during the execution of a process?  Typically no.  A single "document" is being treated by the process.  It may access more data when needed but it would not typically get flooded by other external events.  Do we have to worry about an event overload during the execution of a decision step (business rules)? Typically no.  Business rules apply to a set of business objects.  Additional data may be accessed as needed of course.  I qualified my answers with "typically" because there may be times when many events could happens and influence the processing of a flow or a decision.  Would all those "exceptional paths" belong to the process or business rules.  If limited they may but if we are talking about tons of events we may have to deal with, it is more likely we would want to use a module for processing those events, correlating and filtering them.  Isn't that the forte of CEP?

The CEP / BPM / BRMS world seems fairly simple after all:

• CEP module receives or intercepts a flurry of events and processes them with the objective of figuring out what those events are relevant for; it triggers the appropriate business processes or decision services
• BPM module receives the request for a given process to be applied to a higher level entity (an application, a document...); it automates the steps defined in the business process
• BRMS module is invoked with a given context to apply business rules; it makes a business decision 

Carlos Serrano-Morales wrote a great post on that subject.

Compared to the Human Body

My background is in life sciences, a source of inspiration.  Please indulge me here while I digress in a biological comparison.

I picture CEP as being the sensors.  The skin for example receives tons of inputs (events) on temperature, moisture level, etc.  It is not responsible for making a business decision although it happens that it makes "some level" of decision.  Typically raised hair and goosebumps may result from the sensors detecting cold weather.

BPM would be the nerves carrying the information to the brain and executing on its orders.

As you probably guessed, BRMS would be represented here by the brain, capable of making a higher level decision based on the contextual information correlated by the CEP module (it is cold) but also based on business strategies (use a blanket at night or turn up the heater if that's not enough; get a sweater during the day). 

Assuming that we decide here to use a blanket, the BPM system will be responsible for carrying out this order, get the complete system to get up, walk to the linen closet and grab the most appropriate blanket.

CEP may intervene any time to provide input that may alter the process or the decision making.  If the skin feels cold *and* the eye sees that an ice cube was dropped on the skin, the information would be provided again using the appropriate process / decision service.

I want to emphasize that, although CEP was making some level of decision, it is not built or informed to make the business decision.  There is a clear separation of role.

What makes this so complicated?

I think we are all guilty within the industry.  We are used to use our hammer on any kind of nail or screw in order to feel as ubiquitous as possible.  We hate to see technology used for a specific problem, by fear that it loses appeal.  I actually think that success comes from the acknowledgment that a tool has a purpose, and learning how to use it appropriately.  If we were still trying to convince the world that BRMS is a credible alternative to BPM (and vice versa), we may still be involved in half baked projects (product management recommended book: Angel Customers and Demon Customers).  We would not have been able to grow into a recognized component of the agile architecture.  CEP needs to follow this lead.

I am afraid that we are also getting confused by our own algorithms.  Some Business Rules Engines (BRE) are toying with the usage of Rete to CEP problems.  This should be irrelevant to the role of CEP.  If in the end we use Rete, Parallel Rete or any other algorithm to implement CEP, it should not matter to the role of CEP in the first place.  We must admit that our own Charles Forgy has presented his findings at the latest October Rules Fest. The intention was not to create confusion at the usage level - mea culpa if we did.  We believe that it would be a different Decision Automation module, part of Decision Management of course but not necessarily part of BRMS.

Complication comes from other sources as well.  Because CEP deals with events in time, we sometimes associate it with Business Intelligence (BI), thinking that because it can process events real-time, it should be able to process time-aware query (how many A events happened after a B event).  When combining both aspects (real-time and BI), you could build a Business Activity Monitoring (BAM) system but this does not mean that CEP is BAM either...

Well, as passionate as everyone is on that subject, I am expecting a few heated comments here...  This is healthy as it will help the whole industry as well as the end users to come to a conclusion on what CEP is and what it is good for.  Go ahead!

Comments

[schauecker]

This is a very helpful article and I've been happy to read it. Sometimes a text makes the life easier. ;) just now for me to understand better what the difference between CEP and BPM / BRMS is. thx Carole-Ann.

[Tim Bass]

Dear Carole-Ann,

You are simply (and incorrectly, in my view) reducing CEP to low level event (sensor) processing, which was never the intent of the technology or the architecture.

I covered this in detail in my 8 part series, What is Complex Event Processing?

http://www.thecepblog.com/what-is-complex-event-processing/

and in particular, to analogy to the human body (in Part 1 of 8):

http://www.thecepblog.com/2007/05/14/what-is-complex-event-processing-part-1/

Hope this helps!

Your sincerely, Tim

[Carole-Ann Matignon]

Hi Tim,

I expected some reaction. This is ECA (Event Condition Action) at work ;-)

I am pleased that we actually came to the same human body analogy independently. This validates that we are not so far apart.

Reading the CEP definition posted in your blog, it seems that we mostly differ on one aspect: what you call CEP covers most of what we call Decision Management. I see all sorts of technologies being involved in the Decision Automation world, CEP being the "event processing" part (and just that) and "business rules" being another one (and just that) and "business processes" being yet another one (and just that). All of those plus eventually predictive models (you mention neural nets and Bayesian networks for example in your analogy), optimization, etc. contribute to solving decision making challenges aka Decision Automation.

I personally prefer the term Decision Management to CEP for that ecosystem of technologies. The main reason is that we are not only processing events in there. The focus is purely on managing the decisions we make, whether they are reactive to streams of events or scheduled data processing or yet another trigger in another context. The focus is also on the management of those decision in the sense that business users want to own this logic (business rules or processes), etc. I feel that Decision Management for better or for worse is quite descriptive of the task at hand. It also emphasizes the management part rather than the processing part, and the management part is where the key business value is.

More food in the fire...

Carole-Ann Matignon
VP, Product Management at Fair Isaac

[Carole-Ann Matignon]

And now some water to my well...

I just came across the Oracle announcement of its entry in the CEP world: http://www.ebizq.net/news/10671.html?rss

I quote: "The solution is designed to enable organizations to detect, filter, analyze and correlate diverse business events".

Carole-Ann Matignon
VP, Product Management at Fair Isaac

[Neil]

This reminds me of the story about Henry Kissinger, when Nixon first tapped him to be his National Security Advisor. As he was leaving New York, all of his friends from Columbia (U) threw a party for him. In attendance were some of his Washington friends, too. A journalist, overhearing a very loud conversation going on between some of HK's NY friends asked Kissinger, "Why are arguments in academia so loud, emotional and shrill?" Henry shrugged his shoulders and said, "Because the stakes are so low."

These kinds of definitional arguments are sort of tedious. Why can't there be more than one definition of CEP? Who cares? Marketing people, that's all. The question is, do they work and can they solve a problem that needs to be solved? Why not spend more time articulating that so we can put this stuff to work.

-Neil Raden
twitter: nraden

[Carole-Ann Matignon]

With all due respect Neil, I think you are wrong. We are not picky on this term versus that term but more fundamentally defining what each technology can contribute to the greater good.

If we all let it slip and use our own language without common agreement on what we mean, we will mislead end users and get them to struggle the same way they have struggled when we did not help them enough differentiate BPM from BRMS.

Forrester is working on such a paper too. Knowing that they typically provide those analysis as a result of the questions / demands they get from their customers (which are the real end users), it seems that this exercise has more value than some academic argument on my term versus your term.

Do not underestimate the power of confusion. It lead to decades of dark ages for artificial intelligence. End users hate to be mislead and disappointed. I would not blame them.

Carole-Ann Matignon
VP, Product Management at Fair Isaac

[Tim Bass]

Hi Carole-Ann and Everyone,

I actually agree that it is better to solve real world problems than to argue over descriptive semantics. It is really hard to describe a vast body of knowledge with short catchy phrase like "Complex Event Processing" or "Decision Managment" and, by the very nature of humans, there will be a group, or tribe, which will defend their turf with all the energy they can muster. This is true, not only of marketeers, but of humans in general.

There is certainlhy merit in Carole-Ann's view, but I am not sure I fully agree either; but to be fair, I do not strongly disagree either. I can definately see Carole-Ann's perspective; however, I can see the other view that we are creating artifical boundaries. There is actually nothing wrong with overlap and competition.

I agree that confusion is not good, but it is generally not "confusion in terms" that tends to impede progress as much as the lack of user acceptance. After all, the WWW and HTTP simply work and users like it, because good working solutions always prevail. Endless debates tend to happen when the users do not adopt the technology so the sellers must reposition and try to convince users to adopt.

This is exactly what is happending in the CEP space, IMHO. Folks are not adopting CEP, as envisioned, to "detect business opportunities and threats in real-time" because adding another rules engine type of technology to the solution set(s) does not provide users the additional capabilities they need to solve real-world complex problems. Users already have rules engines, and rules engines are quite mature, from a technology perspective.

The arguments and debates are not coming form the buyers as much as they are coming from the sellers. The buyers have already voted, and they are not buying the hype.

Yours sincerely, Tim

[James Owen]

Carole-Ann:

I fear that I must take umbrage with Mr. Raden. (I have ALWAYS wanted to use that statement and this just seemed like a good time to do it.)

Proper definitions are what gives us a firm foundation upon which to build better and more clearly defined tools and paths from one tool to another, a way to set up boundaries between tools so that one tool cannot claim to be that which it is not, and a way to properly define expected behavior.

An automobile is NOT an SUV is NOT a motorcycle is NOT a bicycle. An SUV is a subset of an automobile while a motorcycle is a subset of a vehicle. These do not, normally, need definition since they are what someone in AI would call a "common sense" understanding of the definition. However, without clear definitions we drift into confusion, chaos and the eventual madness of marketing machinations.

Sorry Neil. You blew it. Maybe your systems should be just a bit smarter. :-)

In contradiction to both you and Carole-Ann, I have never felt that we could equate Complex Event Processing to work flow, business processing (managed or otherwise) nor to biological events. Neural Nets are probably the closest thing that we have to simulating the nerve connections to the brain, a true CEP system. Having worked with the analog versions of such things as well as the binary simulations, I do know that we have nothing today that comes within 10^-15 of the power of the human brain.

Complex Events Processing (CEP) has been defined on Wikipedia as "... primarily an event processing concept that deals with the task of processing multiple events with the goal of identifying the meaningful events within the event cloud. CEP employs techniques such as detection of complex patterns of many events, event correlation and abstraction, event hierarchies, and relationships between events such as causality, membership, and timing, and event-driven processes."

Now, whether we agree or disagree with this definition, we should at LEAST try to come up with an industry agreed-upon definition so that the casual observer would NOT be disillusioned nor led astray by some unscrupulous sales person. Also, there are are many, many more definitions that need standardization in our industries. BRMS was only clarified in 2003 - 2005 because ILOG, Fair Isaac and InfoWorld insisted upon a clear definition.

Now we have Drools coming out with a really strange definition of their version 5 BRMS product which is nothing more than a web interface, repository and GUI all rolled into a single application. Somebody needs to slap their collective wrists for not having taken the time to look around at other definitions before really screwing up the market place with an absolute misnomer.

SDG
jco

[Opher Etzion]

Hi Carole-Ann.

I agree that there is a confusion, and that resolving it is quite simple. See: http://epthinking.blogspot.com/2008/11/on-basic-classification-of-terms.html
for further clarification of this issue.

cheers,

Opher

[Eric Roch]

Interesting tread - related post - mine.

http://it.toolbox.com/blogs/the-soa-blog/eda-cep-bpm-bpms-and-soa-28565

[Carlos Serrano-Morales]

There is no doubt that the industry needs definitions that are clear.

Fuzzy definitions only help technology vendors – who can hope to benefit from the confusion to promote their solution versus the ones from the competition – and the analysts and industry gods and oracles – who can hope to benefit from this very confusion in order to promote their clarification services.

The user community ends with the burden of having to figure out what hides behind the definitions as used by each solution, how to best leverage each one of these solutions, how to combine them, which one to use for which purpose, etc.

Having all the technology pieces provided by a single vendor is no panacea to this problem. These single vendors tend to be large, tend to provide offerings consisting of multiple technologies from varying sources (read acquisitions) that also overlap, and introduce confusion – and I won’t even go into what happens trying to get architectural and technical support.

I think Carole-Ann’s blog is right on target with respect to the definitional boundaries for the clear definition of responsibilities assigned to the various technologies and her analogy is clear.

To elaborate a little, my opinion is that a large part of the debate comes from two key and related issues:
- The domain CEP wants to cover is too wide and overlaps with multiple well established domains
- The CEP terminology is awfully confusing

Carole-Ann covered the first, let me cover the second.

a. “Complex” is absolutely not a good term to use in a definition such as this one.

The definition of “complex event” from Luckham is “It is an event that could only happen if lots of other events happened”.
The problem is that in “Complex Event Processing”, it is often the case that “Complex” is applied to “Processing” more than to “Event”. And this confusion then begs questions such as: “Complex as opposed to what? What is the threshold between simple and complex? Is it the same for all problems and problem areas? Does “complex” describe what the problem to solve is, what we – vendors – need to deal with in the resolution of the problem, or what the user community end up dealing with?”

I do not recall – and I count on the blogosphere to prove me wrong – any such definition that establishes itself using a qualitative adjective such as “complex” at its core.

BRMS, BPM do not qualify their focus in such relative scale: they qualify it on an explicit business objective and a role (or group of roles) they cater to: management of business rules and processes, respectively, and the business management community.

b. “Event processing” is not that good a term either. It’s incredibly wide, in particular because it conveys neither intent nor scope. That creates confusion.
This leads, for example, to the assessment I read somewhere that a Loan Origination application is a CEP application, because it consists of the “complex processing” of the loan application “event”. Of course.. Any request for anything can be converted to the event that the request for that something occurred. And anything you do in response to the event occurrence can be qualified as “complex” (it certainly is for somebody) “processing” (since it probably consists of something else than a NOP).

I do agree with Tim’s comment earlier that it’s not bad to have experimentation and that competition and evolution will bring the best definitions forward. And I would add that is precisely what we are trying to do through this effort...

I do not think the BRMS or BPM industries are protecting their respective turfs in this discussions. We all know products bleed one way or the other. But long gone are the days when BPM vendors would claim they would do BRMS and the contrary – and the industry is better for it.

Just to add to this subject, I posted a little note on IBM's recent BEM announcement in http://architectguy.blogspot.com/2008/12/ibms-bem-announcement.html. More oil on the fire :).