Figure 1. When using traditional middleware technology, we've only been able to create point-to-point links among applications.
Enterprise application integration (EAI) is the latest buzzword from the press and analyst community, even though the concept has been around since the early '70s. EAI is nothing more than the sharing of information among custom or packaged stovepipe applications. Today, EAI lets a new SAP system link to the human resource system and the old mainframe-based inventory application link to the rest of the enterprise systems.
In other words, EAI exposes the data of any application to any other application. The business value of this is obvious.
Truth be told, we've been solving the EAI problem for years using traditional middleware such as message-oriented middleware (MOM), remote procedure calls (RPCs), and distributed objects. When using traditional middleware technology, however, we've only been able to create point-to-point links between applications. (See Figure 1.) Enterprises today look like ill-planned highway systems, each using different technologies to tie applications together and sharing no common infrastructure. Enter message brokers and an opportunity to get application integration under control as well as build on what we already know about middleware.
When you look at what message brokers bring to businesses, the value is clear. A message broker is a technology that can act as a broker among one or more target entities (such as a network, middleware, applications, and systems), letting them share information with less pain than when using traditional techniques (traditional middleware).
Message brokers facilitate the integration of business processes that have until now been more isolated than open. These message brokers use a hub and spoke-type architecture (see Figure 2) and can "broker" information (messages) between any number of entities (applications) no matter how the information is represented or accessed.
To this end, the purpose of a message broker is to integrate multiple business activities (applications) whether they are new, old, legacy, centralized, or distributed. This technology ties together many different platforms and application development solutions, and it can also connect to each application using any number of middleware and API mechanisms as well as route information between them.
In addition to simple routing information, a message broker may provide enhancements such as hosting business functions that build on the existing business functions of the entities they connect. For example, you can build a data extractor for payroll that can also identify potentially fraudulent information coming out of the payroll system. The primary role of a message broker, however, is to provide a simple, central point of integration.
Message brokers often provide data translation engines, which let the message broker change the way information is represented for each application. For instance, and especially considering the Year 2000 problem, the year 1998 may be represented in a legacy system as "98" and in a packaged application as "1998." The same applies to currency data representation within many applications and even the way we represent customer numbers across enterprise applications. Let's examine the concept of the message brokering technology in more detail.
Message brokers leave the systems where they are, minimize changes to them, and still let them share data. This approach provides faster implementation and more flexibility. We are creating middleware that adapts to the applications, not vice versa. A message broker should closely resemble the way businesses actually work, automating functions that are currently being performed manually, such as sending sales reports through interoffice mail or walking data down the hall on disk.
The value of message brokering comes from the idea that traditional stovepipe applications no longer have value by themselves. I define stovepipe applications as self-standing, narrowly focused applications that solve a discrete set of problems. Packaged applications, such as SAP, PeopleSoft, and Baan, seem to become stovepipe applications naturally. In fact, the rapid rise of enterprise resource planning (ERP) drives the need for message brokers because organizations want to hook their new ERP infrastructure up to their traditional bet-the-business apps. No one seems to be cutting over directly to ERP applications - most are integrating and assimilating, and it's taking years, not months.
Other examples of stovepipes are the information systems of another organization acquired through a merger or acquisition or even entities that are part of the supply chain integration problem such as a customer or a supplier. If you look around your organization, I'm sure you'll find many examples of stovepipes, and you'll also find that they are typically not sharing information.
Stovepipes seem to work within their own domain, but executives are no longer accepting the isolation of function that stovepipes cause. When I was consulting, the number-one complaint that I heard from CEOs and CIOs was that they had many valuable applications but none could share information. Moreover, because many of the applications were purchased or created over a 20-year period, they shared almost no common technological frameworks. In other words, they didn't provide obvious points of integration or integrating them was cost prohibitive.
Message brokers, as I already mentioned, are nothing new. We've been creating them in one form or another for years. The message broker vendors, including Active Software Inc., New Era of Networks Software Inc., Century Analysis Inc., and TSI International Software Ltd., are really just turning the concept into products.
In fact, many organizations have spent millions of dollars creating their own custom versions of message brokers, including the U.S. Department of Transportation and Federal Express Corp. They saw the need, did not see the product, and thus created their own message broker to meet a business demand. There are many instances of this; you may already have a first-generation message broker within your organization and not even know it.
Traditional middleware such as MOM only solves part of the problem. Message brokers do not replace such traditional middleware as MOM, RPC, and distributed TP monitors. Message brokers build on top of existing middleware technology, most often on messaging middleware. Messaging middleware is relatively low-level middleware - I've been calling it primitive. Although it provides an effective means of communication among applications, it does little else.
Many components of a business integration solution are neither middleware nor applications. These are typically routing, reformatting, and flow components. These components, which can be placed in an application or in the middleware, are a better architectural fit when placed within message brokers, providing a central point of integration.
So what do message brokers offer? Because the technology is so new, each vendor takes its own approach to solving the message brokering problem. There are some common components, however, including a data translation layer (DTL) and a rules engine. (See Figure 3.)
The DTL understands the formats of all messages being passed among applications and changes the formats on the fly. The data from one message is restructured into a new message that makes better sense to the receiving applications. The DTL provides a common dictionary with information on how each application communicates outside of itself as well as what pieces of information have meaning to what applications.
Typically, DTLs contain parsing and pattern-matching methods that describe the structure of any message format. Message formats are constructed from parts that represent each field encapsulated within a message. Once you break a message down to its component parts, you can recombine the fields to create new messages.
Most brokers can handle most types of messages, including fixed, delimited, and variable formats, although message formats vary widely. You define how to reformat the message using an interface that the message broker provides. This interface may be either as primitive as an API or as easy to use as a GUI.
DTLs typically store the application information in a repository that keeps track of the system where the message is coming from, the format of the message, the target system where the message is routing to, and the desired format of the target system.
If you use a rules engine, you'll be able to implement intelligent message routing and transformation. For example, in some instances a message may be required by one other receiving application. In other instances, you may have to route a message to two or more applications. In order to solve each type of problem you should be able to route the message to any number of applications that are extracting and translating data from any number of applications. This concept is known as many-to-many application integration. To make this concept even more complex, you need to make sure that each application receiving messages is receiving the messages in the proper format.
If you want to create a truly flexible solution, you'll need a rules engine to make these formatting and routing decisions dynamically. A rules engine combined with a DTL makes such dynamic routing and formatting possible, and it lets you define transformation and routing decisions based on the format and content of a message. For example, you know that if a message is in one type of format it's coming from a specific application, and thus you can take the appropriate action such as routing certain messages to another application.
When using rules engines, you're creating an application between applications that does nothing more than provide the logic for sharing information among the applications. Each rules engine is going to solve the intelligent routing and translation problem differently, but most provide the capability to test a message to determine fields and values. Most often these rules engines use traditional Boolean logic (such as if, else, and or) and high-level languages to create rules as well as associate actions to each rule if found to be true. For example:
IF SAP: Inventory Database State Change; Extract Transaction Data; Reformat Transaction Data for Human_Resources; Update Human_Resources; END;
Is this the next generation of middleware? I think so. Message brokers fill a need that many larger companies have and that traditional middleware won't solve. The downside is that you'll have to go through the normal confusion that arises when a new technology attempts to find its place within the enterprise. I've looked at most message brokering products out there, and they all take very different approaches to solving the same set of problems. It may be a good idea for those guys to get on the same sheet of music. Promoting the concept of a message broker - at least at first - could be more beneficial than promoting a particular product.
The upside is that we are finally stepping up to the plate and using good technology to solve a growing problem that's already been driving CIOs nuts for years. It seems that in the process of creating all those client/server, distributed, and host-based applications, we were solving problems without providing solutions. It's time that we get to this very important task.
Figure 1. When using traditional middleware technology, we've only been able to create point-to-point links among applications.
Figure 2. Message brokers use a hub and spoke-type architecture and are able to "broker" information (messages) between any number of entities.
Figure 3. Common components of message brokers include a data translation layer, a rules engine, and sometimes a set of appication adapters.