Messaging:
An enterprise has multiple applications that are being built independently, with different languages and platforms. The enterprise needs to share data and processes in a responsive way.
How can I integrate multiple applications so that they work together and can exchange information?
Use Messaging to transfer packets of data frequently, immediately, reliably, and asynchronously, using customizable formats.
Asynchronous messaging is fundamentally a pragmatic reaction to the problems of distributed systems. Sending a message does not require both systems to be up and ready at the same time. Furthermore, thinking about the communication in an asynchronous manner forces developers to recognize that working with a remote application is slower, which encourages design of components with high cohesion (lots of work locally) and low adhesion (selective work remotely).
Message Channel
An enterprise has two separate applications that need to communicate, preferably by using Messaging.
How does one application communicate with another using messaging?
Connect the applications using a Message Channel, where one application writes information to the channel and the other one reads that information from the channel.
When an application has information to communicate, it doesn't just fling the information into the messaging system, it adds the information to a particular Message Channel. An application receiving information doesn't just pick it up at random from the messaging system; it retrieves the information from a particular Message Channel.
P1 Category:
1) Message
An enterprise has two separate applications that are communicating via Messaging, using a Message Channel that connects them.
How can two applications connected by a message channel exchange a piece of information?
Package the information into a Message, a data record that the messaging system can transmit through a message channel.
Thus any data that is to be transmitted via a messaging system must be converted into one or more messages that can be sent through messaging channels.
2) Command Message
An application needs to invoke functionality provided by other applications. It would typically use Remote Procedure Invocation, but would like to take advantage of the benefits of using Messaging.
How can messaging be used to invoke a procedure in another application?
Use a Command Message to reliably invoke a procedure in another application.
There is no specific message type for commands; a Command Message is simply a regular message that happens to contain a command. In JMS, the command message could be any type of message; examples include an ObjectMessage containing a Serializable command object, a TextMessage containing the command in XML form, etc. In .NET, a command message is a Message with a command stored in it. A Simple Object Access Protocol (SOAP) request is a command message.
3) Document Message
An application would like to transfer data to another application. It could do so using File Transfer or Shared Database, but those approaches have shortcomings. The transfer might work better using Messaging.
How can messaging be used to transfer data between applications?
Use a Document Message to reliably transfer a data structure between applications.
Whereas a Command Message tells the receiver to invoke certain behavior, a Document Message just passes data and lets the receiver decide what, if anything, to do with the data. The data is a single unit of data, a single object or data structure which may decompose into smaller units.
4) Event Message
Several applications would like to use event-notification to coordinate their actions, and would like to use Messaging to communicate those events.
How can messaging be used to transmit events from one application to another?
Use an Event Message for reliable, asynchronous event notification between applications.
When a subject has an event to announce, it will create an event object, wrap it in a message, and send it on a channel. The observer will receive the event message, get the event, and process it. Messaging does not change the event notification, just makes sure that the notification gets to the observer.
5) Request- Reply
When two applications communicate via Messaging, the communication is one-way. The applications may want a two-way conversation.
When an application sends a message, how can it get a response from the receiver?
Send a pair of Request-Reply messages, each on its own channel.
Request-Reply has two participants:
6) Return Address
My application is using Messaging to perform a Request-Reply.
How does a replier know where to send the reply?
The request message should contain a Return Address that indicates where to send the reply message.
This way, the replier does not need to know where to send the reply, it can just ask the request. If different messages to the same replier require replies to different places, the replier knows where to send the reply for each request. This encapsulates the knowledge of what channels to use for requests and replies within the requestor so those decisions do not have to be hard coded within the replier. A Return Address is put in the header of a message because it’s not part of the data being transmitted.
...
Example: Golang Channels (NEW)
In Go, channels are first-class language constructs, making it easy to specify a channel as return address by including a field of type chan in the structure to be passed as the request:
type Request struct {
data []int
resultChan chan int
}
A function handling requests can now publish responses to the channel provided in the Request structure:
func handle(queue chan *Request) {
for req := range queue {
req.resultChan <- sum(req.data)
}
}
Requestors can now specify the desired return channel with the request as in the following example:
reqChannel := make(chan *Request, 10)
go handle(reqChannel)
// Make two requests with separate return channels
request1 := &Request{[]int{3, 4, 5}, make(chan int)}
request2 := &Request{[]int{1, 2, 3}, make(chan int)}
reqChannel <- request1
reqChannel <- request2
// Receive results on respective channels
fmt.Printf("answer: %d %d\n", <-request1.resultChan, <-request2.resultChan)
Find the source for this code snippet on Github
7) Correlation Identifier
My application is using Messaging to perform a Request-Reply and has received a reply message.
How does a requestor that has received a reply know which request this is the reply for?
Each reply message should contain a Correlation Identifier, a unique identifier that indicates which request message this reply is for.
There are six parts to Correlation Identifier:
...
8) Message Sequence
My application needs to send a huge amount of data to another process, more than may fit in a single message. Or my application has made a request whose reply contains too much data for a single message.
How can messaging transmit an arbitrarily large amount of data?
Whenever a large set of data may need to be broken into message-size chunks, send the data as a Message Sequence and mark each message with sequence identification fields.
The three Message Sequence identification fields are:
9) Message Expiration
My application is using Messaging. If a Message’s data or request is not received by a certain time, it is useless and should be ignored.
How can a sender indicate when a message should be considered stale and thus shouldn’t be processed?
Set the Message Expiration to specify a time limit how long the message is viable.
Once the time for which a message is viable passes, and the message still has not been consumed, then the message will expire. The messaging system’s consumers will ignore an expired message; they treat the message as if it where never sent in the first place. Most messaging system implementations reroute expired messages to the Dead Letter Channel, while others simply discard expired messages; this may be configurable.
...
10) Message Expiration
My application is using Messaging. If a Message’s data or request is not received by a certain time, it is useless and should be ignored.
How can a sender indicate when a message should be considered stale and thus shouldn’t be processed?
Set the Message Expiration to specify a time limit how long the message is viable.
Once the time for which a message is viable passes, and the message still has not been consumed, then the message will expire. The messaging system’s consumers will ignore an expired message; they treat the message as if it where never sent in the first place. Most messaging system implementations reroute expired messages to the Dead Letter Channel, while others simply discard expired messages; this may be configurable.
11) Format Indicator
Several applications are communicating via Messages that follow an agreed upon data format, perhaps an enterprise-wide Canonical Data Model. However, that format may need to change over time.
How can a message’s data format be designed to allow for possible future changes?
Design a data format that includes a Format Indicator, so that the message specifies what format it is using.
The format indicator enables the sender to tell the receiver the format of the message. This way, a receiver expecting several possible formats knows which one a message is using and therefore how to interpret the message’s contents.
P2: Message Routing
1) Pipes and Filters
In many enterprise integration scenarios, a single event triggers a sequence of processing steps, each performing a specific function. For example, let's assume a new order arrives in our enterprise in the form of a message. One requirement may be that the message is encrypted to prevent eavesdroppers from spying on a customer's order. A second requirement is that the messages contain authentication information in the form of a digital certificate to ensure that orders are placed only by trusted customers. In addition, duplicate messages could be sent from external parties (remember all the warnings on the popular shopping sites to click the 'Order Now' button only once?). To avoid duplicate shipments and unhappy customers, we need to eliminate duplicate messages before subsequent order processing steps are initiated. To meet these requirements, we need to transform a stream of possibly duplicated, encrypted messages containing extra authentication data into a stream of unique, simple plain-text order messages without the extraneous data fields.
How can we perform complex processing on a message while maintaining independence and flexibility?
Use the Pipes and Filters architectural style to divide a larger processing task into a sequence of smaller, independent processing steps (Filters) that are connected by channels (Pipes).
Each filter exposes a very simple interface: it receives messages on the inbound pipe, processes the message, and publishes the results to the outbound pipe. The pipe connects one filter to the next, sending output messages from one filter to the next. Because all component use the same external interface they can be composed into different solutions by connecting the components to different pipes. We can add new filters, omit existing ones or rearrange them into a new sequence -- all without having to change the filters themselves. The connection between filter and pipe is sometimes called port. In the basic form, each filter component has one input port and one output port.
2) Message Router
Multiple processing steps in a Pipes and Filters chain are connected by Message Channels.
How can you decouple individual processing steps so that messages can be passed to different filters depending on a set of conditions?
Insert a special filter, a Message Router, which consumes a Message from one Message Channel and republishes it to a different Message Channel channel depending on a set of conditions.
The Message Router differs from the most basic notion of Pipes and Filters in that it connects to multiple output channels. Thanks to the Pipes and Filters architecture the components surrounding the Message Router are completely unaware of the existence of a Message Router. A key property of the Message Router is that it does not modify the message contents. It only concerns itself with the destination of the message.
...
3) Content-Based Router
Assume that we are building an order processing system. When an incoming order is received, we first validate the order and then verify that the ordered item is available in the warehouse. This function is performed by the inventory system. This sequence of processing steps is a perfect candidate for the Pipes and Filters style. We create two filters, one for the validation step and one for the inventory system, and route the incoming messages through both filters. However, in many enterprise integration scenarios more than one inventory system exists with each system being able to handle only specific items.
How do we handle a situation where the implementation of a single logical function (e.g., inventory check) is spread across multiple physical systems?
Use a Content-Based Router to route each message to the correct recipient based on message content.
The Content-Based Router examines the message content and routes the message onto a different channel based on data contained in the message. The routing can be based on a number of criteria such as existence of fields, specific field values etc. When implementing a Content-Based Router, special caution should be taken to make the routing function easy to maintain as the router can become a point of frequent maintenance. In more sophisticated integration scenarios, the Content-Based Router can take on the form of a configurable rules engine that computes the destination channel based on a set of configurable rules.
...
Example: Apache Camel (NEW)
Many open source ESBs like Apache Camel have direct support for many Enterprise Integration Patterns, wither in a Java-based DSL or as configuration. For example, setting up a Content-Based Router in Camel is quite easy and idiomatic by using a RouteBuilder:
RouteBuilder builder = new RouteBuilder() {
public void configure() {
errorHandler(deadLetterChannel("mock:error"));
from("direct:in")
.choice()
.when(header("type").isEqualTo("widget"))
.to("direct:widget")
.when(header("type").isEqualTo("gadget"))
.to("direct:gadget")
.otherwise()
.to("direct:other");
}
};
4) Message Filter
Continuing with the order processing example, let's assume that company management publishes price changes and promotions to large customers. Whenever a price for an item changes, we send a message notifying the customer. We do the same if we are running a special promotion, e.g. all widgets are 10% off in the month of November. Some customers may be interested in receiving price updates or promotions only related to specific items. If I purchase primarily gadgets, I may not be interested in knowing whether widgets are on sale or not.
How can a component avoid receiving uninteresting messages?
Use a special kind of Message Router, a Message Filter, to eliminate undesired messages from a channel based on a set of criteria.
The Message Filter has only a single output channel. If the message content matches the criteria specified by the Message Filter, the message is routed to the output channel. If the message content does not match the criteria, the message is discarded.
...
Example: RabbitMQ Bindings (NEW)
RabbitMQ allows a straightforward implementation of Publish-Subscribe Channel with Message Filters via a direct exchange. Exchanges are a concept which is distinct from a queue. Generally, message producers publish messages to an exchange, which controls how the message is propagated to other queues. For example, a "fanout" exchange simply propagates a message to all subscribed queues while a "direct" exchange allows subscribing queues to express a filter condition for the messages they want to receive.
A direct exchange places incoming messages into a subscribing queue based on a message's routing key and the queue's binding key. Multiple binding keys are allowed per queue and multiple queues can use the same binding key, which results in each of them receiving a copy of the queue.
Mapping RabbitMQ Concepts to Enterprise Integration Patterns
The Widgets and Gadgets example from above can be implemented quite easily with a single direct exchange and two queues bound to this exchange, each with its respective filter. A helper method helps create a new queue, bind it to the exchange with the filter, and setup a Event-Driven Consumer that simply prints out messages.
static void filteredReceive(final Channel channel, String filter, String exchangeName) throws IOException {
System.out.println(filter + " waiting for messages. To exit press CTRL+C");
Consumer consumer = new DefaultConsumer(channel) {
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body)
throws IOException {
String message = new String(body, "UTF-8");
System.out.println(filter + " Received '" + message + "'");
}
};
String queueName = channel.queueDeclare().getQueue();
channel.queueBind(queueName, EXCHANGE_NAME, filter);
boolean autoAck = true;
channel.basicConsume(queueName, autoAck, consumer);
}
The consumer is a simple Event-Driven Consumer that is called every time a message is received. The following lines generate a new private queue and bind it to the exchange with the specified filter string. Lastly, the consumer implementation is attached to the queue.
We can now create the direct exchange and start multiple of these consumers, giving each a respective binding key:
channel.exchangeDeclare(EXCHANGE_NAME, "direct"); filteredReceive(channel, "Widget", EXCHANGE_NAME); filteredReceive(channel, "Gadget", EXCHANGE_NAME);
Lastly, we publish some messages against the exchange with different message keys:
channel.exchangeDeclare(EXCHANGE_NAME, "direct"); sendMessage(channel, EXCHANGE_NAME, "Widget", "Hello EIP Widget!"); sendMessage(channel, EXCHANGE_NAME, "Gadget", "Hello EIP Gadget!");
sendMessage is a simple helper method:
static void sendMessage(Channel channel, String exchange, String key, String message) throws IOException {
channel.basicPublish(exchange, key, null, message.getBytes());
System.out.println(" [x] Sent '" + message + "'");
}
You can run this code and observe that RabbitMQ filters the messages according to their key:
Widget waiting for messages. To exit press CTRL+C Gadget waiting for messages. To exit press CTRL+C Widget Received 'Hello EIP Widget!' Gadget Received 'Hello EIP Gadget!'
The complete source code for this example can be found on Github.
5) Dynamic Router
You are using a Message Router to route messages between multiple destinations.
How can you avoid the dependency of the router on all possible destinations while maintaining its efficiency?
Use a Dynamic Router, a Router that can self-configure based on special configuration messages from participating destinations.
Besides the usual input and output channels the Dynamic Router uses an additional control channel. During system start-up, each potential recipient sends a special message to the Dynamic Router on this control channel, announcing its presence and listing the conditions under which it can handle a message. The Dynamic Router stores the 'preferences' for each participant in a rule base. When a message arrives, the Dynamic Router evaluates all rules and routes the message to the recipient whose rules are fulfilled. This allows for efficient, predictive routing without the maintenance dependency of the Dynamic Router on each potential recipient.
...
6) Recipient List
A Content-Based Router allows us to route a message to the correct system based on message content. This process is transparent to the original sender in the sense that the originator simply sends the message to a channel, where the router picks it up and takes care of everything.
In some cases, though, we may want to specify one or more recipients for the message. A common analogy are the recipient lists implemented in most e-mail systems. For each e-mail message, the sender can specify a list of recipients. The mail system then ensures transport of the message content to each recipient. An example from the domain of enterprise integration would be a situation where a function can be performed by one or more providers. For example, we may have a contract with multiple credit agencies to assess the credit worthiness of our customers. When a small order comes in we may simply route the credit request message to one credit agency. If a customer places a large order, we may want to route the credit request message to multiple agencies and compare the results before making a decision. In this case, the list of recipients depends on the dollar value of the order.
In another situation, we may want to route an order message to a select list of suppliers to obtain a quote for the requested item. Rather than sending the request to all vendors, we may want to control which vendors receive the request, possibly based on user preferences
How do we route a message to a list of dynamically specified recipients?
Define a channel for each recipient. Then use a Recipient List to inspect an incoming message, determine the list of desired recipients, and forward the message to all channels associated with the recipients in the list.
The logic embedded in a Recipient List can be pictured as two separate parts even though the implementation is often coupled together. The first part computes a list of recipients. The second part simply traverses the list and sends a copy of the received message to each recipient. Just like a Content-Based Router, the Recipient List usually does not modify the message contents.
...
7) Splitter
Many messages passing through an integration solution consist of multiple elements. For example, an order placed by a customer consists of more than just a single line item. As outlined in the description of the Content-Based Router, each line item may need to be handled by a different inventory system. Thus, we need to find an approach to process a complete order, but treat each order item contained in the order individually.
How can we process a message if it contains multiple elements, each of which may have to be processed in a different way?
Use a Splitter to break out the composite message into a series of individual messages, each containing data related to one item.
use a Splitter that consumes one message containing a list of repeating elements, each of which can be processed individually. The Splitter publishes a one message for each single element (or a subset of elements) from the original message.
8) Aggregator
A Splitter is useful to break out a single message into a sequence of sub-messages that can be processed individually. Likewise, a Recipient List or a Publish-Subscribe Channel is useful to forward a request message to multiple recipients in parallel in order to get multiple responses to choose from. In most of these scenarios, the further processing depends on successful processing of the sub-messages. For example, we want to select the best bid from a number of vendor responses or we want to bill the client for an order after all items have been pulled from the warehouse.
How do we combine the results of individual, but related messages so that they can be processed as a whole?
Use a stateful filter, an Aggregator, to collect and store individual messages until a complete set of related messages has been received. Then, the Aggregator publishes a single message distilled from the individual messages.
The Aggregator is a special Filter that receives a stream of messages and identifies messages that are correlated. Once a complete set of messages has been received (more on how to decide when a set is 'complete' below), the Aggregator collects information from each correlated message and publishes a single, aggregated message to the output channel for further processing
9) A Message Router can route messages from one channel to different channels based on message content or other criteria. Because individual messages may follow different routes, some messages are likely to pass through the processing steps sooner than others, resulting in the messages getting out of order. However, some subsequent processing steps do require in-sequence processing of messages, for example to maintain referential integrity.
How can we get a stream of related but out-of-sequence messages back into the correct order?
Use a stateful filter, a Resequencer, to collect and re-order messages so that they can be published to the output channel in a specified order.
The Resequencer can receive a stream of messages that may not arrive in order. The Resequencer contains in internal buffer to store out-of-sequence messages until a complete sequence is obtained. The in-sequence messages are then published to the output channel. It is important that the output channel is order-preserving so messages are guaranteed to arrive in order at the next component. Like most other routers, a Resequencer usually does not modify the message contents.
10) Composed Message Processor
The order-processing example presented in the Content-Based Router and Splitter patterns processes an incoming order consisting of individual line items. Each line item requires an inventory check with the respective inventory system. After all items have been verified we want to pass the validated order message to the next processing step.
How can you maintain the overall message flow when processing a message consisting of multiple elements, each of which may require different processing?
Use Composed Message Processor to process a composite message. The Composed Message Processor splits the message up, routes the sub-messages to the appropriate destinations and re-aggregates the responses back into a single message.
The Composed Message Processor uses an Aggregator to reconcile the requests that were dispatched to the multiple inventory systems. Each processing unit sends a response message to the aggregator stating the inventory on hand for the specified item. The Aggregator collects the individual responses and processes them based on a predefined algorithm as described under Aggregator.
...
11) Scatter-Gather
In the order processing example introduced in the previous patterns , each order item that is not currently in stock could be supplied by one of multiple external suppliers. However, the suppliers may or may not have the respective item in stock themselves, they may charge a different price and may be able to supply the part by a different date. To fill the order in the best way possible, we should request quotes from all suppliers and decide which one provides us with the best term for the requested item.
How do you maintain the overall message flow when a message needs to be sent to multiple recipients, each of which may send a reply?
Use a Scatter-Gather that broadcasts a message to multiple recipients and re-aggregates the responses back into a single message.
The Scatter-Gather routes a request message to the a number of recipients. It then uses an Aggregator to collect the responses and distill them into a single response message.
|
12) Routing Slip
Most of the routing patterns presented in this section route incoming messages to one or more destinations based on a set of rules. Sometimes, though, we need to route a message not just to a single component, but through a whole series of components. Let's assume, for example, that we use a Pipes and Filters architecture to process incoming messages that have to undergo a sequence of processing steps and business rule validations. Since the nature of the validations varies widely and may depend on external systems (e.g., credit card validations), we implement each type of step as a separate filter. Each filter inspects the incoming message, and applies the business rule(s) to the message. If the message does not fulfill the conditions specified by the rules it is routed to an exception channel. The channels between the filters determine the sequence of validations that the message needs to undergo.
Now let's assume, though, that the set of validations to perform against each message depends on the message type (for example, purchase order request do not need credit card validation or customers who send orders over a VPN may not require decryption and authentication). To accommodate this requirement we need to find a configuration that can route the message through a different sequence of filters depending on the type of the message.
How do we route a message consecutively through a series of processing steps when the sequence of steps is not known at design-time and may vary for each message?
Attach a Routing Slip to each message, specifying the sequence of processing steps. Wrap each component with a special message router that reads the Routing Slip and routes the message to the next component in the list.
We insert a special component into the beginning of the process that computes the list of required steps for each message. It then attaches the list as a Routing Slip to the message and starts the process by routing the message to the first processing step. After successful processing, each processing step looks at the Routing Slip and passes the message to the next processing step specified in the routing table.
13) Process Manager
The Routing Slip demonstrates how a message can be routed through a dynamic series of processing steps. The solution of the Routing Slip is based on two key assumptions: the sequence of processing steps has to be determined up-front and the sequence is linear. In many cases, these assumptions may not be fulfilled. For example, routing decisions might have to be made based on intermediate results. Or, the processing steps may not be sequential, but multiple steps might be executed in parallel.
How do we route a message through multiple processing steps when the required steps may not be known at design-time and may not be sequential?
Use a central processing unit, a Process Manager, to maintain the state of the sequence and determine the next processing step based on intermediate results.
First of all, let me clarify that the design and configuration of a Process Manager is a pretty extensive topic. We could probably fill a whole book (Volume 2, maybe?) with patterns related to the design of workflow or business process management. Therefore, this pattern is intended primarily to "round off" the topic of routing patterns and to provide a pointer into the direction of workflow and process modeling. By no means is it a comprehensive treatment of business process design.
14) Message Broker
Many patterns in this chapter present ways to route messages to the proper destination without the originating application being aware of the ultimate destination of the message. Most of the patterns focused on specific types of routing logic. However, in aggregate, these patterns solve a bigger problem.
How can you decouple the destination of a message from the sender and maintain central control over the flow of messages?
Use a central Message Broker that can receive messages from multiple destinations, determine the correct destination and route the message to the correct channel. Implement the internals of the Message Broker using the design patterns presented in this chapter.
Using a central Message Broker is sometimes referred to as hub-and-spoke architectural style, which appears to be a descriptive name when looking at the diagram above.
P3: Message Transformation
1) Message Translator
The previous patterns describe how to construct messages and how to route them to the correct destination. In many cases, enterprise integration solutions route messages between existing applications such as legacy systems, packaged applications, homegrown custom applications, or applications operated by external partners. Each of these applications is usually built around a proprietary data model. Each application may have a slightly different notion of the Customer entity , the attributes that define a Customer and which other entities a Customer is related to. For example, the accounting system may be more interested in the customer's tax payer ID numbers while the customer-relationship management (CRM) system stores phone numbers and addresses. The application’s underlying data model usually drives the design of the physical database schema, an interface file format or a programming interface (API) -- those entities that an integration solution has to interface with. As a result, the applications expect to receive messages that mimic the application's internal data format.
In addition to the proprietary data models and data formats incorporated in the various applications, integration solutions often times interact with standardized data formats that seek to be independent from specific applications. There are a number of consortia and standards bodies that define these protocols, such as RosettaNet, ebXML, OAGIS and many other, industry specific consortia. In many cases, the integration solution needs to be able to communicate with external parties using the ‘official’ data formats while the internal systems are based on proprietary formats.
How can systems using different data formats communicate with each other using messaging?
Use a special filter, a Message Translator, between other filters or applications to translate one data format into another.
The Message Translator is the messaging equivalent of the Adapter pattern described in [GoF]. An adapter converts the interface of a component into a another interface so it can be used in a different context.
...
2) Envelope Wrapper
Most messaging systems divide the message data into a header and a body. The header contains fields that are used by the messaging infrastructure to manage the flow of messages. However, most endpoint systems that participate in the integration solution generally are not aware of these extra data elements. In some cases, systems may even consider these fields as erroneous because they do not match the message format used by the application. On the other hand, the messaging components that route the messages between the applications may require the header fields and would consider a message invalid if it does not contain the proper header fields.
How can existing systems participate in a messaging exchange that places specific requirements on the message format, such as message header fields or encryption?
Use a Envelope Wrapper to wrap application data inside an envelope that is compliant with the messaging infrastructure. Unwrap the message when it arrives at the destination.
The process of wrapping and unwrapping a message consists of five steps:
3) Content Enricher
When sending messages from one system to another it is common for the target system to require more information than the source system can provide. For example, incoming Address messages may just contain the ZIP code because the designers felt that storing a redundant state code would be superfluous. Likely, another system is going to want to specify both a state code and a ZIP code field. Yet another system may not actually use state codes, but spell the state name out because it uses free-form addresses in order to support international addresses. Likewise, one system may provide us with a customer ID, but the receiving system actually requires the customer name and address. An order message sent by the order management system may just contain an order number, but we need to find the customer ID associated with that order, so we can pass it to the customer management system. The scenarios are plentiful.
How do we communicate with another system if the message originator does not have all the required data items available?
Use a specialized transformer, a Content Enricher, to access an external data source in order to augment a message with missing information.
The Content Enricher uses information inside the incoming message (e.g. key fields) to retrieve data from an external source. After the Content Enricher retrieves the required data from the resource, it appends the data to the message. The original information from the incoming message may be carried over into the resulting message or may no longer be needed, depending on the specific needs of the receiving application.
4) Content Filter
How do you simplify dealing with a large message, when you are interested only in a few data items?
Use a Content Filter to remove unimportant data items from a message leaving only important items.
The Content Filter does not necessarily just remove data elements. A Content Filter is also useful to simplify the structure of the message. Often times, messages are represented as tree structures. Many messages originating from external systems or packaged applications contain many levels of nested, repeating groups because they are modeled after generic, normalized database structures. Frequently, known constraints and assumptions make this level of nesting superfluous and a Content Filter can be used to 'flatten' the hierarchy into a simple list of elements than can be more easily understood and processed by other systems.
5) Claim Check
The Content Enricher tells us how we can deal with situations where our message is missing required data items. The Content Filter lets us remove uninteresting data items from a message. Sometimes, we want to remove fields only temporarily. For example, a message may contain a set of data items that may be needed later in the message flow, but that are not necessary for all intermediate processing steps. We may not want to carry all this information through each processing step because it may cause performance degradation and makes debugging harder because we carry so much extra data.
How can we reduce the data volume of message sent across the system without sacrificing information content?
Store message data in a persistent store and pass a Claim Check to subsequent components. These components can use the Claim Check to retrieve the stored information.
The Claim Check pattern consists of the following steps:
6) Normalizer
In a business-to-business (B2B) integration scenario it is quite common for an enterprise to receive messages from different business partners. These message may have the same meaning, but follow different formats, depending on the partners' internal systems and preferences. For example, we built a solution for a pay-per-view provider that has to accept and process viewership information from over 1700 (!) affiliates, most of which did not conform to a standard format.
How do you process messages that are semantically equivalent, but arrive in a different format?
Use a Normalizer to route each message type through a custom Message Translator so that the resulting messages match a common format.
The Normalizer uses a Message Router to route the incoming message to the correct Message Translator. This requires the Message Router to detect the type of the incoming message. Many messaging systems equip each message with a type specifier field in the Message Header to make this type of task simple. However, in many B2B scenarios messages do not arrive as messages compliant with the enterprise's internal messaging system, but in diverse formats such as comma separated files or XML document without associated schema. While it is certainly best practice to equip any incoming data format with a type specifier we know all to well that the world is far from perfect. As a result, we need to think of more general ways to identify the format of the incoming message. One common way for schema-less XML documents is to use the name of the root element to assume the correct type. If multiple data formats use the same root element, you can use XPATH expressions to determine the existence of specific sub-nodes. Comma-separated files can require a little more creativity. Sometimes you can determine the type based on the number of fields and the type of the data (e.g. numeric vs. string). If the data arrives as files, the easiest way may be to use the file name or the file folder structure as a surrogate Datatype Channel. Each business partner can name the file with a unique naming convention. The Message Router can then use the file name to route the message to the appropriate Message Translator.
...
7) Canonical Data Model
I am designing several applications to work together through Messaging. Each application has its own internal data format.
How can you minimize dependencies when integrating applications that use different data formats?
Therefore, design a Canonical Data Model that is independent from any specific application. Require each application to produce and consume messages in this common format.
The Canonical Data Model provides an additional level of indirection between application's individual data formats. If a new application is added to the integration solution only transformation between the Canonical Data Model has to created, independent from the number of applications that already participate.
P4: Messaging Endpoints
1) Message Endpoint
Applications are communicating by sending Messages to each other via Message Channels.
How does an application connect to a messaging channel to send and receive messages?
Connect an application to a messaging channel using a Message Endpoint, a client of the messaging system that the application can then use to send or receive messages.
Message Endpoint code is custom to both the application and the messaging system’s client API. The rest of the application knows little about message formats, messaging channels, or any of the other details of communicating with other applications via messaging. It just knows that it has a request or piece of data to send to another application, or is expecting those from another application. It is the messaging endpoint code that takes that command or data, makes it into a message, and sends it on a particular messaging channel. It is the endpoint that receives a message, extracts the contents, and gives them to the application in a meaningful way.
2) Messaging Gateway
An application accesses another system via Messaging.
How do you encapsulate access to the messaging system from the rest of the application?
Use a Messaging Gateway, a class than wraps messaging-specific method calls and exposes domain-specific methods to the application.
The Messaging Gateway encapsulates messaging-specific code (e.g., the code required to send or receive a message) and separates it from the rest of the application code. This way, only the Messaging Gateway code knows about the messaging system; the rest of the application code does not. The Messaging Gateway exposes a business function to the rest of the application so that instead of requiring the application to set properties like Message.MessageReadPropertyFilter.AppSpecific, a Messaging Gateway exposes methods such as GetCreditScore that accept strongly typed parameters just like any other method. A Messaging Gateway is a messaging-specific version of the more general Gateway pattern [EAA].
3) Messaging Mapper
When integrating applications using messaging, the data inside a message is often derived from domain objects inside the integrated applications. If we use a Document Message, the message itself may directly represent one or domain objects. If we use a Command Message, some of the data fields associated with the command are likely to be extracted from domain objects as well. There are some distinct differences between messages and objects. For example, most objects rely on associations in the form of object references and inheritance relationships. Many messaging infrastructures do not support these concepts because they have to be able to communicate with a range of applications, some of which may not be object-oriented at all.
How do you move data between domain objects and the messaging infrastructure while keeping the two independent of each other?
Create a separate Messaging Mapper that contains the mapping logic between the messaging infrastructure and the domain objects. Neither the objects nor the infrastructure have knowledge of the Messaging Mapper's existence.
The Messaging Mapper accesses one or more domain objects and converts them into a message as required by the messaging channel. It also performs the opposite function, creating or updating domain objects based on incoming messages. Since the Messaging Mapper is implemented as a separate class that references the domain object(s) and the messaging layer, neither layer is aware of the other. The layers don't even know about the Messaging Mapper.
4) Transactional Client
A messaging system, by necessity, uses transactional behavior internally. It may be useful for an external client to be able to control the scope of the transactions that impact its behavior.
How can a client control its transactions with the messaging system?
Use a Transactional Client—make the client’s session with the messaging system transactional so that the client can specify transaction boundaries.
Both a sender and a receiver can be transactional. With a sender, the message isn’t “really” added to the channel until the sender commits the transaction. With a receiver, the message isn’t “really” removed from the channel until the receiver commits the transaction. A sender that uses explicit transactions can be used with a receiver that uses implicit transactions, and vise versa. A single channel might have a combination of implicitly and explicitly transactional senders; it could also have a combination of receivers.
...
Example: (Not quite) Transactional Client in Amazon SQS (NEW)
Transactional Client semantics can cause serious performance degradation in widely distributed systems. Many cloud-based messaging systems therefore don't implement Transactional Clients. However, these systems still aim to provide protection against client s crashing after they consume but before they process a message. The Amazon Simple Queuing Service (SQS) provides this protection via so-called Visibility Timeout. In this configuration, message consumers don't actually remove messages from the queue - the message simply becomes invisible to other consumers. After successfully processing the message, a message consumer must explicitly delete the message from the queue. If a message is not deleted, it becomes visible again to other consumers after the set timeout interval expires.
The book Cloud Computing Patterns documents this behavior as a distinct pattern called Time-out Based Delivery.
5)
5)
No comments:
Post a Comment