IIB best practices

Introduction

Best Practices

---

Understand the Interface Business Requirement

---

·        The first key to successful Technical Design and Interface Development is a firm knowledge of the intended business logic and structure

·        The source to the Business Requirement is to acquire a firm and defined Functional Specification. To some degree, the Functional Specification may undergo a few minor amendments while the task of Technical Specification is underway. However, it is expected that the Functional Specification has defined all the messages, components and processes that are required for the interface before the commencement of Technical Design

·        There may be (and usually is) multiple components to each interface, and in IIB terms, that boils down to Adapters and Message Flows

·        The Functional Specification is used to define where the Business Requirements are met in terms of various components. It is required to consider the various options available and create a design that is based on workable functionality of IIB, and to hopefully end up with a reasonable level of maintainability and performance as well

Common Control Handling

---

·        The flow of each message through a IIB interface should be 'Control Handled' for audit and logging purpose. For this reason, there is a requirement to have a 'Control Mechanism' in each and every message flow.

·        It is recommended to have a 'Common Control Handling' routine/component that can be re-used extensively by all interfaces of the project.

Common Characteristics of Interfaces

---

·        Each IIB Interface designed for the project should have certain characteristics, which are assumed to be common across all implemented interfaces

·        These common characteristics generally refer to some common routines and flows or sub-flows, which are recommended to be used globally across all interfaces in that project

Common Error/Exception Handling

---

·        Each IIB interface shall have a mechanism to handle errors/exceptions. These errors/exceptions refer to both Systems generated ones as well as User Generated ones.

·        It is recommended to have a common Error Handling Procedure/Module to handle all errors/exceptions that arise during execution of an interface.

·        This Procedure/Module shall be extensively re-used by all interfaces of the project.

Interface design Layout

---

·        The IIB interfaces to be implemented should have certain common characteristics so that reuse of components is optimized

·        The design of the message flows should adhere to one or more of the pattern templates laid out for the project

Common Routines and Sub-Flows

---

Location of Integration Logic

---

·        An interface is comprised of Adapters and Message Flows

·        The integration logic is therefore contained wholly within the adapters and message flows

·        The interface designer has the option of building any required integration logic in the adapters or message flows

·        It is recommended that as much business logic as possible will be performed in message flows rather than in the adapters. In essence logic is only to be located in adapters if that is the only option for the interface.

·        In other words, adapters will me made as simple as possible. The core of the business logic should be kept in WMB.

·        The interface solutions are pattern based, with the patterns represented by WMB message flows. Being pattern based, enhances reusability.

·        Scaling and resilience is simplified with fewer complex components to consider.

·        Maintenance requirements are easier to define and execute, by largely avoiding the need to change one component to accommodate the change in another requirement.

Understand the Input and Output Messages

---

Determine how many Message Flows are needed

---

·        First, at an application system level, it must be determined how many flows will be required for an interface. Out of this determination, it will be possible to determine how many types of messages a flow must handle.

·        Sometimes there is just one input message format and one output message format. Other times there are multiple messages for a given flow and there must be logic within the flow to sort them out.

Message Routing within a flow

---

·        Routing within the flow should be minimized where possible. When routing is required, then the rule about minimizing the number of nodes still applies.

·        For a number of reasons, some message flows process more than one message type. In this case it is important to understand the distribution of these different message types and their importance so that the most critical or most frequent messages go through the cheapest path; i.e. the one with fewest and simplest nodes.

·        When more than one message is processed in a flow, or when there is a requirement for dynamic logic inside a flow based on message content, there are at least two ways to direct the processing logic inside the flow.

·        One way is through the use of RouteToLabel node. This node makes use of the Destination List array of the Local Environment Tree of a message flow. When the message arrives, the Destination List is populated according to the business requirement, and the RouteToLabel and Label nodes are used to direct subsequent processing.

Complete the processing with as few nodes as possible

---

·        The most fundamental recommendation to consider when writing a message flow is to be as concise as possible. There is a performance cost associated with passing through each of the built-in nodes of IIB, so it is best to write message flows in as few nodes as possible.

·        In other words, there should be as few as possible nodes between the input and output formats for a given business requirement.

Sub-Flows

---

·        Message Flows can be divided into SubFlows. Essentially, a Subflow is a sequence of nodes that begins with an Input node and ends in an Output node. The object of using SubFlows is that functions can be logically segregated in a way that increases maintainability and offers opportunity for reuse.

Transactionality and Persistence

---

·        As a rule, all messages should be handled as a Transaction.

·        In the Input, Output and Compute nodes, the Transactionality should always be set to ‘Automatic’. This practice allows the WebSphere MQ message attributes to control the Transactionality of the flow; i.e. if a message is read from a queue with an attribute of ‘Persistent’, IIB will supply the appropriate logging to guarantee the Transactionality of the message.

·        It must be considered that overhead is incurred by making a message transactional. This is caused by the need to save all the data necessary to enable a roll back should a failure condition occur in a message flow.

Use of CARDINALITY

---

ESQL REFERENCE Variable

---

·        Reference variables can be used in ESQL to store a part or an entire message tree location and can be thought of as being similar to a C/C++ pointer to a message element. The use of REFERENCE variable is recommended in ESQL for two reasons:

·        They reduce coding considerably, especially, when large message tree with deep hierarchy has been defined.

·        They make ESQL more efficient when used appropriately.

·        Reference variables can be used to store locations in any type of message tree (InputRoot, OutputRoot, Local Environment, Environment, etc).

Backout Processing Considerations

---

·        When an exception is thrown within a message flow and is not caught by inclusion of a TryCatch Node, The Input Node for the message flow catches it. If the catch terminal of the MQInput Node is connected, the message is propagated to this terminal and is processed according to the message flow logic.

·        If the Catch terminal of the MQInput Node is not connected, the transaction is rolled back. If the message was read under a syncpoint the original remains in the queue but WebSphere MQ increments the MQMD Backoutcount.

·        The MQInput Node then reads the message once again.

·        The MQMD Backoutcount is examined before the message is processed again. If it is not zero then it implies that the message received by the message flow is a backed out message and the broker then performs backout processing.

·        If the MQMD backoutcount is less than the Backout Requeue Threshold attribute specified in the queue definition, the message is propagated through the output terminal of the MQInput Node for normal processing once again.

·        If the MQMD backoutcount is not less than the Backout Requeue Threshold attribute specified in the queue definition, the message is propagated through the Failure terminal of the MQInput Node.

·        If the Failure terminal is wired then it follows that path but, if it is not wired the Backout Requeue Name attribute is looked for and if a queue name found in this attribute, the message is put into that queue.

·        If no name is specified in the Backout Requeue Name attribute, the message is written to the default Dead Letter Queue defined for the queue manager.

·        If the message could not be written to the Dead Letter Queue then it remains in the Input Queue.

IIB Message Flow Standard Unit Test Conditions

---

·        The following are the standard Unit Test Conditions that should be considered by a developer while performing Unit Test on an interface:

·        When a valid Test Data is passed through the message flow one or more output message (s) are created on the output queue (s).

·        When an invalid Test Data is passed through the message flow the message is propagated to the error handling queue for error processing.

·        All fields of the output message are of correct length and have been correctly formatted according to the requirements.

·        Where the message has repeating fields or structure, the interface works correctly for both single and multiple instances of the field or structure.

·        Where the message has an optional field or structure, the interface works correctly for both with and without the field or structure in the message.

Important Structures

---

MQMD – Message Descriptor

---

Field	Description
StrucId	Structure Identifier
Version	Structure Version Number
Report	Option for report messages
MsgType	Message Type
Expiry	Message Lifetime
Feedback	Feedback or Reason Code
Encoding	Numeric Encoding of message Data
CodedCharSetId	Character set identifier of message data
MsgId	Message Identifier
User Identifier	User Identifier
AccountingToken	Accounting Token
ApplIdentityData	Application data relating to identity
PutApplType	Type of application that put the message
PutApplName	Name of application that put the message
PutDate	Date when message was put

Field	Description
Format	Format name of message data
Priority	Priority of message data
Persistence	Message Persistence
MsgSeqNumber	Sequence number of logical message within group
CorrelId	Correlation Identifier
BackoutCount	BackoutCounter
ReplyToQ	Name of Reply Queue
ReplyToQMgr	Name of Reply Queue Manager
PutTime	Time when message was put
ApplOriginData	Application Data relating to origin
GroupId	Group Identifier
Offset	Offset of data in physical message from the start of logical message
MsgFlags	Message Flags
Original Length	Length of original message

MQRFH2 – Rules and Formatting Header

---

Field	Description
StrucId	Structure Identifier
Version	Structure Version Number
StrucLength	The total length of MQRFH2
Encoding	Numeric encoding of data
CodedCharSetId	Character set identifier of data
Format	Format Name of data
Flags	Flags
NameValueCCSID	Character set identifier of Name Value data
NameValueLength	Length of Name Value data
NameValueData	Name/Value data

Properties – Message Flow

---

Field	Description
MessageSet	The Message Set Identifier
MessageType	The Message Name
MessageFormat	The Physical format of the message
Encoding	Numeric encoding of data
CodedCharSetId	Character set identifier of data
Transactional	Boolean flag to indicate Transactionality
Persistence	Boolean flag to indicate Persistency
CreationTime	The put time of the message
ExpirationTime	The expiration time of the message
Priority	The priority of the message
ReplyIdentifier	The reply identifier
ReplyProtocol	The reply protocol

Debug Tracing

---

·        Open a command prompt window.

·        Execute the command mqsichangetrace -u -e -l debug -f -r

·        Execute the message flow with the message that you want to trace.

·        Execute the command mqsireadlog -u -e -l debug -f -o

·        Execute the command mqsiformatlog –i -o

·        Open the generated Trace files in a text editor and perform diagnosis.

x

This document presents some findings and recommendations concerning the use of IBM Integration Bus. These findings were developed from previous experiences of various integration projects. It covers the aspects of IIB development encompassing Message Set guidelines, Message Flow guidelines and Deployment guidelines.

Many message flows require to perform certain functions like re-formatting a date, getting current date, getting a unique identifier, etc which can be laid down as common routines for re-use by the interfaces. Also there may be certain processes like adding an RFH2 header to a message, etc which can also be laid down as common sub-flows for re-use by interfaces.

It is recommended to have a set of common routines and sub-flows for the project that can be extensively re-used by the interfaces. The contents of this set cannot be defined exhaustively at any point of time. This set shall continually be gaining members from time to time in order to meet requirements.

The benefits of this approach are:

It should be possible to examine the Functional Specification and determine what the input is, and what the output would be. From this standpoint, the WMB message flow can be regarded as the instrument to convert the input message into output message(s).

The use of the CARDINALITY function should be restricted to minimum as it requires the parser to process a large portion of the message and thereby hindering performance. This built-in function should not be used inside a loop unless it is inevitable.

The MQMD structure contains the control information that accompanies the application data when a message travels from one queue to another.

The MQRFH2 structure defines the layout of the rules and formatting header. This header can be used to send string data that has been encoded using an XML-like syntax. 

It allows Unicode string to be transported without translations, and it can carry numeric data-types.

The following table summarizes the fields in the structure:

The Properties tree is the first element of the message tree and holds information about the characteristics of the message.

In the event of an error in a message flow, a debug level trace should be set up for the flow. This is done with the help of a set of commands to generate the debug trace file for diagnostics.

Auto Logon Python

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

usernameStr = 'userid@gmail'
passwordStr = 'pwd'

browser = webdriver.Chrome()
browser.get(('https://accounts.google.com/signin/v2/identifier?service=mail&passive=true&rm=false&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F&ss=1&scc=1&ltmpl=default&ltmplcache=2&emr=1&osid=1&flowName=GlifWebSignIn&flowEntry=ServiceLogin'))
# fill in username and hit the next button
username = browser.find_element_by_id('Sign in')
username.send_keys(usernameStr)
nextButton = browser.find_element_by_id('next')
nextButton.click()

# wait for transition then continue to fill items
password = WebDriverWait(browser, 1).until(
    EC.presence_of_element_located((By.ID, 'Passwd')))
password.send_keys(passwordStr)

signInButton = browser.find_element_by_id('signin')
signInButton.click()


Mouse

import pyautogui import win32api import sys pyautogui.FAILSAFE = False for i in range(80000000000000000000): if win32api.GetAsyncKeyState(ord('K')): sys.exit() pyautogui.moveTo(1, 1, duration=1.25) pyautogui.click(10, 5) pyautogui.moveTo(100, 100, duration=1.25)

REST Vs SOAP

REST (REpresentational State Transfer) is an architectural style for developing web services. REST is popular due to its simplicity and the fact that it builds upon existing systems and features of the internet's HTTP in order to achieve its objectives, as opposed to creating new standards, frameworks and technologies.

Advantages of REST

A primary benefit of using REST, both from a client and server's perspective, is REST-based interactions happen using constructs that are familiar to anyone who is accustomed to using the internet's Hypertext Transfer Protocol (HTTP).

An example of this arrangement is REST-based interactions all communicate their status using standard HTTP status codes. So, a 404 means a requested resource wasn't found; a 401 code means the request wasn't authorized; a 200 code means everything is OK; and a 500 means there was an unrecoverable application error on the server.

Similarly, details such as encryption and data transport integrity are solved not by adding new frameworks or technologies, but instead by relying on well-known Secure Sockets Layer (SSL) encryption and Transport Layer Security (TLS). So, the entire REST architecture is built upon concepts with which most developers are already familiar.

REST is also a language-independent architectural style. REST-based applications can be written using any language, be it Java, Kotlin, .NET, AngularJS or JavaScript. As long as a programming language can make web-based requests using HTTP, it is possible for that language to be used to invoke a RESTful API or web service. Similarly, RESTful web services can be written using any language, so developers tasked with implementing such services can choose technologies that work best for their situation.

The other benefit of using REST is its pervasiveness. On the server side, there are a variety of REST-based frameworks for helping developers create RESTful web services, including RESTlet and Apache CXF. From the client side, all of the new JavaScript frameworks, such as JQuery, Node.js, Angular and EmberJS, all have standard libraries built into their APIs that make invoking RESTful web services and consuming the XML- or JSON-based data they return a relatively straightforward endeavor.

Disadvantages of REST

The benefit of REST using HTTP constructs also creates restrictions, however. Many of the limitations of HTTP likewise turn into shortcomings of the REST architectural style. For example, HTTP does not store state-based information between request-response cycles, which means REST-based applications must be stateless and any state management tasks must be performed by the client.

Similarly, since HTTP doesn't have any mechanism to send push notifications from the server to the client, it is difficult to implement any type of services where the server updates the client without the use of client-side polling of the server or some other type of web hook.

From an implementation standpoint, a common problem with REST is the fact that developers disagree with exactly what it means to be REST-based. Some software developers incorrectly consider anything that isn't SOAP-based to be RESTful. Driving this common misconception about REST is the fact that it is an architectural style, so there is no reference implementation or definitive standard that will confirm whether a given design is RESTful. As a result, there is discourse as to whether a given API conforms to REST-based principles.

Alternatives to REST

Alternate technologies for creating SOA-based systems or creating APIs for invoking remote microservices include XML over HTTP (XML-RPC), CORBA, RMI over IIOP and the Simple Object Access Protocol (SOAP).

Each technology has its own set of benefits and drawbacks, but the compelling feature of REST that sets it apart is the fact that, rather than asking a developer to work with a set of custom protocols or to create a special data format for exchanging messages between a client and a server, REST insists the best way to implement a network-based web service is to simply use the basic construct of the network protocol itself, which in the case of the internet is HTTP.

This is an important point, as REST is not intended to apply just to the internet; rather, its principles are intended to apply to all protocols, including WEBDAV, FTP and so on.

REST vs. SOAP

The two competing styles for implementing web services are REST and SOAP. The fundamental difference between the two is the philosophical approach the two have to remotely invocations.

REST takes a resource-based approach to web-based interactions. With REST, you locate a resource on the server, and you choose to either update that resource, delete it or get some information about it.

With SOAP, the client doesn't choose to interact directly with a resource, but instead calls a service, and that service mitigates access to the various objects and resources behind the scenes.

SOAP has also built a large number of frameworks and APIs on top of HTTP, including the Web Services Description Language (WSDL), which defines the structure of data that gets passed back and forth between the client and the server.

Some problem domains are served well by the ability to stringently define the message format, or can benefit from using various SOAP-related APIs, such as WS-Eventing, WS-Notification and WS-Security. There are times when HTTP cannot provide the level of functionality an application might require, and in these cases, using SOAP is preferable. 

REST URIs and URLs

Most people are familiar with the way URLs and URIs work on the web. A RESTful approach to developing applications asserts that requesting information about a resource should be as simple as invoking its URL.

For example, if a client wanted to invoke a web service that listed all of the quizzes available here at TechTarget, the URL to the web service would look something like this:

www.techtarget.com/restfulapi/quizzes

When invoked, the web service might respond with the following JSON string listing all of the available quizzes, one of which is about DevOps:

{ "quizzes" : [ "Java", "DevOps", "IoT"] }

To get the DevOps quiz, the web service might be called using the following URL:

www.techtarget.com/restfulapi/quizzes/DevOps

Invoking this URL would return a JSON string listing all of the questions in the DevOps quiz. To get an individual question from the quiz, the number of the question would be added to the URL. So, to get the third question in the DevOps quiz, the following RESTful URL would be used:

www.techtarget.com/restfulapi/quizzes/DevOps/3

Invoking that URL might return a JSON string such as the following:

{ "Question" : {"query":"What is your DevOps role?", "optionA":"Dev", "optionB":"Ops"} }

As you can see, the REST URLs in this example are structured in a logical and meaningful way that identifies the exact resource being requested.

JSON and XML REST data formats

The example above shows JSON used as the data exchange format for the RESTful interaction. The two most common data exchange formats are JSON and XML, and many RESTful web services can use both formats interchangeably, as long as the client can request the interaction to happen in either XML or JSON.

Note that while JSON and XML are popular data exchange formats, REST itself does not put any restrictions on what the format should be. In fact, some RESTful web services exchange binary data for the sake of efficiency. This is another benefit to working with REST-based web services, as the software architect is given a great deal of freedom in terms of how best to implement a service.

REST and the HTTP methods

The example above only dealt with accessing data.

The default operation of HTTP is GET, which is intended to be used when getting data from the server. However, HTTP defines a number of other methods, including PUT, POST and DELETE.

The REST philosophy asserts that to delete something on the server, you would simply use the URL for the resource and specify the DELETE method of HTTP. For saving data to the server, a URL and the PUT method would be used. For operations that are more involved than simply saving, reading or deleting information, the POST method of HTTP can be used.

History of REST

REST was first coined by computer scientist Roy Fielding in his year-2000 Ph.D. dissertation at the University of California, titled Architectural Styles and the Design of Network-based Software Architectures.

Chapter 5 of the dissertation, "Representational State Transfer (REST)," described Fielding's beliefs about how best to architect distributed hypermedia systems. Fielding noted a number of boundary conditions that describe how REST-based systems should behave. These conditions are referred to as REST constraints, with four of the key constraints described below:

• Use of a uniform interface (UI). As stated earlier, resources in REST-based systems should be uniquely identifiable through a single URL, and only by using the underlying methods of the network protocol, such as DELETE, PUT and GET with HTTP, should it be possible to manipulate a resource.
• Client-server-based. In a REST-based system, there should be a clear delineation between the client and the server. UI and request-generating concerns are the domain of the client. Meanwhile, data access, workload management and security are the domain of the server. This separation allows loose coupling between the client and the server, and each can be developed and enhanced independent of the other.
• Stateless operations. All client-server operations should be stateless, and any state management that is required should happen on the client, not the server.
• RESTful resource caching. The ability to cache resources between client invocations is a priority in order to reduce latency and improve performance. As a result, all resources should allow caching unless an explicit indication is made that it is not possible.