ESQL code to create mail with attachments using broker events

In this post I will provide an example on how to process events generated by a flow using the default IBM Integration Bus monitoring event capability.

The example will show how

• to serialize a tree into BLOB using ESQL
• how to send a mail with attachment
• How to create/prepare the LocalEnvironment and Root for the EmailOutput node
• The usage of the business event capabilities provided by IIB

The principle is simple:

1. Configure a flow to generate an IIB event
2. Create a subscription to this event with a WMQ Queue as endpoint
3. Create a flow that consumes these events and send an email with attachments

The example in this post shows how to create mail with attachments using ESQL but this could be easily made using Java as well.

Configure a flow to generate an IIB event

The event generated as a well defined structure and the schema can be imported into a library using new model -> IBM predefined model.

Any nodes in a flow can be configured to generate events (Generating events in WebSphere Message Broker) that may contain context information and payload.

It is for example possible to configure a node to include the localEnvironment, ExceptionList and Message tree structure (under Root). These information will be placed into the IIB events under the folder "complexContent".
Note that the LocalEnvironment is reset when an exception occurs, so the data that would have been stored in this tree would be wiped when the message is propagated to the catch terminal of the input node (will be covered in a future post).

Finally it is also possible to include the full payload (as it was received) by selecting in the monitoring node properties "include payload as bitstream". The payload will then be included into the IIB events under "BistreamData".

Create a subscription

The IIB runtime is publishing the IIB events on the WMQ topic "$SYS/Broker/IBMIBus/Monitoring/#".

Using the WMQ Explorer you create a subscription to these events and select a destination queue:

 The flow that sends email

The flow is very simple: MQInput -> ComputeNode -> EmailOutput node

The compute node is used to create and configure the message that will be send using the emailoutput node. 

The node it self is configured the minimum properties: server:port, email to, from and security.

The rest will be provided by the code in ESQL (subject, body content and attachments).

In this example the complexContent included in the incoming business event is serialized into bitstream and will be send by mail as attachment.

The payload if present is also send as attachment.

The body of the mail is made of event origin data and using a DFDL to have a text document separated with CRLF.

The code is provided here after:

	CALL CopyMessageHeaders();
	-- CALL CopyEntireMessage();
	/*
	emailoutput node requires the following (with attachments):
	- LocalEnvironment->Destination->Email->Attachment
	-- Attachment -> Content (BLOB)/ContentType (CHAR)/ContentName(CHAR)/ContentEncoding (CHAR)
	- Root -> EmailOutput -> Subject (if you would like to be overriden at runtime
	- Root -> BLOB (this is the actual body of the mail
	*/
	-- Create the LocalEnvironment folders
	DECLARE ptrMail REFERENCE TO OutputLocalEnvironment;
	CREATE LASTCHILD OF ptrMail AS ptrMail TYPE Name NAME 'Destination';
	CREATE LASTCHILD OF ptrMail AS ptrMail TYPE Name NAME 'Email';
	DECLARE ptrAttachment REFERENCE TO ptrMail;
	-- Get the information from the broker events
	DECLARE ptrEvents REFERENCE TO InputRoot.XMLNSC.ns:event.ns:applicationData.ns:complexContent;
	WHILE LASTMOVE(ptrEvents) DO
	CREATE LASTCHILD OF ptrMail AS ptrAttachment TYPE Name NAME 'Attachment';
	SET OutputRoot.XMLNSC.Data = ptrEvents;
	-- Content has to be BLOB to be taken as an attachment.
	-- Serialize the broker event payload available as complexContent
	SET ptrAttachment.Content = ASBITSTREAM(OutputRoot.XMLNSC.Data OPTIONS FolderBitStream);
	SET ptrAttachment.ContentType = 'text/xml';
	-- Specify the attanchment name
	SET ptrAttachment.ContentName = ptrEvents.ns:elementName;
	-- Set as binary
	SET ptrAttachment.ContentEncoding = 'binary';
	DELETE FIELD OutputRoot.XMLNSC;
	-- move to the next complexContent
	MOVE ptrEvents NEXTSIBLING REPEAT NAME;
	END WHILE;
	MOVE ptrEvents TO InputRoot.XMLNSC.ns:event.ns:bitstreamData.ns:bitstream;
	IF LASTMOVE(ptrEvents) THEN
	-- If there is the bitstream available, send it over as attachment
	CREATE LASTCHILD OF ptrMail AS ptrAttachment TYPE Name NAME 'Attachment';
	--bitstream is base64 encoded, so first we need to decode it
	-- ** the node allows to encode the data in Base64, it doesn't allows to provide a already base64 data though
	SET ptrAttachment.Content =
	BASE64DECODE(InputRoot.XMLNSC.ns:event.ns:bitstreamData.ns:bitstream);
	SET ptrAttachment.ContentType = 'text/plain';
	SET ptrAttachment.ContentName = 'payload';
	-- ask the node to convert it into Base64
	SET ptrAttachment.ContentEncoding = 'Base64';
	END IF;
	-- Set the subject
	SET OutputRoot.EmailOutputHeader.Subject = 'Mail Service info';
	-- In order to have a pretty message layout, a DFDL is used with CRLF terminator.
	SET OutputRoot.Properties.MessageFormat = 'DFDL';
	SET OutputRoot.Properties.MessageType = '{}:mailMessage';
	SET OutputRoot.DFDL.mailMessage.Header.MessageType = 'INFO';
	MOVE ptrEvents TO InputRoot.XMLNSC.ns:event.ns:eventPointData.ns:messageFlowData;
	SET OutputRoot.DFDL.mailMessage.Header.Origin = ptrEvents.ns:node.ns:nodeLabel || ptrEvents.ns:node.ns:terminal;
	SET OutputRoot.DFDL.mailMessage.Header.Server = ptrEvents.ns:messageFlow.ns:uniqueFlowName;
	SET OutputRoot.DFDL.mailMessage.Header.MessageId = InputRoot.XMLNSC.ns:event.ns:eventPointData.ns:eventData.ns:eventCorrelation.ns:localTransactionId;
	SET OutputRoot.DFDL.mailMessage.Messages.Message = 'simple content';
	-- Serialize the DFDL into BLOB for the OutputMail node.
	-- This will be the mail body.
	SET OutputRoot.BLOB.BLOB = ASBITSTREAM(OutputRoot.DFDL.mailMessage OPTIONS RootBitStream);
	DELETE FIELD OutputRoot.DFDL;
	RETURN TRUE;

IBM Integration Explorer installation

I recently had some trouble with my IIB plugin (administration) after applying a MQ Explorer Fix:
I downloaded the MQX fix, and after the installation, the IIB nodes are not available anymore in my explorer.

It seems that the IIB is not installed.
I remembered that the plugin is configured by adding the links in the eclipse installation folder.
This seems to be ok as well.

I decided to uninstall the IBX and reinstall it but even that doesn't solve my issue.

After chatting with some Gurus, I finally discovered that eclipse does not notice that the plugin is no more linked or updated.

So what is the option?

The trick is to uninstall the IBX and reinstall it but under another directory.
From that moment, I usually set a version number on the folder name:
C:\IBM\IIBExplorerFP2

This is it.

Don't forget to start the IBX after using the -c option.

If you still have any trouble, let me know

Test SSL configuration with curl

In a previous article I explained how to configure IBM Integration Bus to use HTTPS with an InputHTTP node (httphttps-listener-behavior-with-iib.html).

I realized that it may not be evident to test the configuration.
I will provide here some hints on how to test a configuration where a server (for instance IBM Integration Bus) is configured to receive https connections with mutual authentication.

For these test I am using very useful tools: curl and openSSL.
useful information on curl can be found here.

Configuration

The keystore of the Integration Server node holds the personal certificate of the server. This certificate contains a public and private key.

The trustStore of the Integration Server node holds the certificate of the client that needs to be authenticated. This certificate contains only a public key. This key has been provided by the client.

In order to make a mutual authentication, the public key of the server has to be provided to the client and the client has to provide it's public key.

To extract the IBM Integration Server certificate from the JKS keystore, one can use the IBM Key Man tool. This tool is started from the menu (windows) or using strmqikm.

Select the folder Personal Certificate and click on extract certificate. If you select the type as "Base64-encoded ASCII data" the certificate will be in the PEM format (privacy-enhanced mail).

The tool provides as extension "arm". This format is equivalent to pem. The extension can be changed from PEM to ARM.

If you used the key man tool to create a self-signed certificate for the client, you would need to export the certificate in pkcs12 format (p12). This certificate would contains the public and private key.

Certificate format 

PEM  (privacy-enhanced mail) format
It's a "Base64-encoded ASCII data" certificate and this format is equivalent to arm. The extension can be changed from PEM to ARM.
PKSC12 certificate may have pfx or p12 as extension.
DER is a binary encoded certificate.
Get more information on SSLShopper

Testing

Curl requires PEM certificate.

In our example here, the client needs to have a personal certificate to be able to sign. The personal certificate for the client is in PKCS12 format, therefore you would need to convert it in PEM format.

This conversion can be done using openSSL (openssl commands) using the command:

openssl pkcs12 -in ClientPersonalCert.p12 -out ClientPersonalCert.pem -nodes

You can then use Curl to call the service.

curl --carcert serverCertificate.pem --cert clientPersonalCert.pem:<password> --cert-type PEM https://myserver:port/test

• serverCertificate.pem is the certificate from the server that has been extracted from the keystore. It holds only a public key
• clientPersonalCert.pem is the personal certificate of the client. This certificate has been exported from a keystore and has been converted into a PEM format.

If you need to perform an HTTP GET with query parameters, you may use the following curl command:

curl --carcert serverCertificate.pem --cert clientPersonalCert.pem:<password> --cert-type PEM -G -d "<myqueryParms>" https://myserver:port/test

• G is to tells that you are issuing a GET
• d is to provides the query parameters

HTTP/HTTPS listener behavior with IIB HTTPInput nodes

HTTP/HTTPS listener behavior with IIB HTTPInput nodes

When a flow containing HTTPInput nodes are deployed on an Integration Server, the default behavior is to use the broker wide HTTP Listener.

This is different is you are deploying a flow using SOAP nodes. In this later case, the http listener used is the embedded HTTP listener of the Integration Server.

For your information, the broker wide listener is using MQ behind the scene. So it can be enabled on the version 10 if a default queue manager has not been configured.

In this blog I will explain how to configure the Integration Node to use the embedded listener of an Integration Server when using HTTP nodes. 

I will also explain how to configure the Integration Node to use SSL (HTTPS).

In the following text, I will assume that

* The integration node is called: IBMIBus

* The integration server is called: IServer1

Configuration for Embedded HTTP Listener

First check the configuration of the Integration Server using the following command:

mqsireportproperties IBMIBus -e IServer1 -o ExecutionGroup -a

This command will show the property "httpNodesUseEmbeddedListener". If this property is set to true, this means that when you will deploy a flow having a HTTPInput node, the embedded HTTP listener will be used.

To change this value use the following command:

mqsichangeproperties IBMIBus-e IServer1 -o ExecutionGroup -n httpNodesUseEmbeddedListener -v true

The port used by the embedded HTTP listener is defined dynamically when the first flow having HTTP nodes is deployed or when the Integration Server is started if it had such flow already deployed. If no flow having HTTP nodes has been deployed, the listener will not be activated.

To check the port used by the embedded HTTP listener, use the following command:

mqsireportproperties IBMIBus -e IServer1 -o HTTPConnector -a

The port can be specified if required (this will disable the automatic port number attribution). This is done using the following command:

mqsichangeproperties IBMIBus -e IServer1 -o HTTPConnector -n explicitlySetPortNumber -v 8085

Embedded listener configuration for SSL (HTTPS)

In this part, I will provide the commands to configure the embedded HTTP listener to use SSL.

Prerequisites

* The Integration Server has been configured to use embedded HTTP listener

* A key store has been created. It contains a certificate for the integration server (that can be used for the public and private key)

* A key store or trust store containing the client certificate if mutual authentication is required.

* The password used to access the keystore is "password".

The keystore and truststore configuration can be found at the following link:

Setting up a public key infrastructure

Configuration

The Integration Server uses two objects to configure the SSL: the ComIbmJVMManager and the HTTPSConnector

The ComIBMJVMManager object is used for the entire Integration Server. It is used by input HTTP nodes as well as request HTTP nodes.

The HTTPSConnector is used only for the input HTTP nodes. 

If you need different keystore for the http request nodes and for the http input nodes then you may configure the ComIBMJVMManager for the HTTP request nodes and the HTTPSConnector for the input http node.

If there is no differences, you can configure only the ComIBMJVMManager object.

ComIBMJVMManager configuration

The following command is used to configure the object:

mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n keystoreFile -v "c:\ks_IBMIBus.jks"
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n truststoreFile -v "c:\ks_IBMIBus.jks"
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n keystorePass -v <password>
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n truststorePass -v <password>
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n keystoreType -v JKS
mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n truststoreType -v JKS

<password> the password to provide. You may provide the password directly in the command line or store the password in the secure integration node registry using the command mqsisetdbparms.

To use the secure registry, you have to provide the password in the command line with the form <MyIntegrationServer>Keystore::password. The command would then be:

mqsichangeproperties IBMIBus -e IServer1 -o ComIbmJVMManager -n keystorePass -v IServer1Keystore::password

Then store the password using the command line

mqissetdbparms IBMIBus -n IServer1Keystore::password -u ignore -p password

The user has no usage here, you may set whatever value you would like.

You need to restart the integration node if you change any of these properties.

If you need to configure the HTTPSConnector, follow the same approach.

Important Note: if you are using a browser tools like HttpRequestor from firefox, you would first need to accept the server certificate. This may be done by simply performing a GET of the service URL in firefox self. You would then be prompted to accept the certificate.

Specific server certificate to be used

You can specify the certificate to be used by the HTTPInput node for SSL. By default the first personal certificate found in the keystore is used. This certificate is used to authenticate the server to the client.

If you require to set a specific one set the property "keyAlias" of the object HTTPSConnector to the right alias.

mqsichangeproperties IBMIBus -e IServer1 -o HTTPSConnector -n keyAlias -v myAlias

Mutual authentication

To enable mutual authentication,  the property "clientAuth" of the object HTTPSConnector has to be set to true.

mqsichangeproperties IBMIBus -e IServer1 -o HTTPSConnector -n clientAuth -v true

By setting this value you would have using a browser:

Error code: ssl_error_handshake_failure_alert

Create a certificate and add the certificate containing the public/private key to the browser and the public certificate to the Integration Server Truststore (or keystore depending of your configuration). 

On firefox, this is done by going to option -> Advanced -> Certificates -> View Certificates -> Your Certificate -> import

You should have a pfx or p12 file ready.

You may create a self signed certificate for test, using the IBM key Management tool. 

Create a self signed certificate then export and select the "PKCS12" key file type.

Referenceable Variable in API Connect

This topic provides information about variables that can be defined and or accessed at runtime in API Connect.

These variables exists in the API Connect environment during the execution of the APIC assembly.
The assembly is the definition of the processing flow that is executed by the API Gateway at runtime when a request is received. This processing flow consists of policies and these can be "validate", "map", switch, javascript, xslt, invoke, ...

These variable can be referenced in APIC policies using the notation $(varName). For example in a invoke policy, you might want to evaluate dynamically the host name or the URL:

In this snapshot, the "backendurl" and the "payid" are variable that are referenced.

What are the variable that can be defined and referenced?

There are two main type of variables: context variable and API properties.
Context variables are variables that the value usually depends of the message request whereas API properties have their values defined when the API is published.

There are three main subcategories (my interpretation) of context variables:

• user variables: these variables are defined in the assembly using javascript, xslt, or set variable
• policies variables: these variables are instantiated by the policy that you set on the assembly. For example the validate policy creates context variable "decoded.claims" (validate jwt).
• apic environment variables: these variables are set by the apic environment. For example the variable message (corresponds to the current data on the flow) and the request (corresponds to the input message and should be considered immutable).

How can these variables be accessed/defined ?

API Properties

API properties are defined by the "properties" in the API design UI (or the yaml). The value of these variables may differ depending on which catalog the API is published. 

This can be useful if for example within the same APIC cloud environment different catalogs are defined for different environment (f.e. DEV, TEST). You can for example define a "backendurl" that would be different if the API is deployed in the DEV catalog or in the TEST catalog.

These variable are accessed at runtime the same way as context variable using the notation $(myPropertyVar) or in a gatewayscript/xslt using the getvariable function (more later).

Context Variables

Unless set by the runtime, context variables are set programatically using either the setVariable, the gatewayscript (javascript) or the xslt policy.

In gatewayscript or xslt the variables are accessed using a provided getVariable or setVariable function.

Using gatewayscript:

apim.setvariable('payid',myvar,'set');
apim.getvariable('paid');

or using xslt:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:dp="http://www.datapower.com/extensions"
  xmlns:func="http://exslt.org/functions"
  xmlns:apim="http://www.ibm.com/apimanagement" extension-element-prefixes="dp func apim">
  <!-- Contains the APIM functions -->
  <xsl:import href="local:///isp/policy/apim.custom.xsl" />
  <xsl:template match="/">
    <xsl:call-template name="apim:setVariable">
      <xsl:with-param name="varName" select="'payid'" />
      <xsl:with-param name="value" select="'avalue'" />
    </xsl:call-template>
    <xsl:message>
      <xsl:text>Variable [</xsl:text>
      <xsl:value-of select="'payid'" />
      <xsl:text>] set to [</xsl:text>
      <xsl:value-of select="'avalue'" />
      <xsl:text>]</xsl:text>
    </xsl:message>
  </xsl:template>
</xsl:stylesheet>

More example can be found here for gatewayscript or for xslt.

Available runtime context variable

A list of runtime context variable that you can access are provided here at the knowledge center.

Notes that even when the API Gateway is DataPower, only the API runtime context variable can be reached. You can't for example access DataPower service variable.
If you would like to access them, you would need to create an APIC custom policy that would use either a gatewayscript action or xsl action to set an APIC environment variable using the apim set variable (using as example the snippet code provided before).

Websphere IIB record and replay

You will find in this post the commands that is necessary to configure the Integration Bus to record and replay the events generated by the flows.

Even though all the following information is available in the knowledge center, I found sometime difficult to find out all the necessary commands to be executed.

Parameters

Integration server

Integration Node name: <INName>
Integration Server name: <ISName>
IIB Queue Manager name: <IIBQMgrName>

Configurable Service

    Data capture store name. Configuration to define the database to be used: <DCStoreN>
DataCaptureSource name. Configuration to define the event source: <DCSourceN>
Data Destination Name: configuration used to define the queue where the message will be send when using the replay mechanism: <DDName>
Queue name used to send back the data: <ReplayQName>

Database configuration

ODBC Database DSN: <DSN>
Table Schema used to store the IIB events: <IIBSchema>
User/password for accessing the database under the schema <IIBSchema>: <DBUsr>/<BDPwd>

Script

Create configurable services for IIB

1. DataCaptureSource
 

mqsicreateconfigurableservice <INName> -c DataCaptureSource -o <DCSourceN> -n dataCaptureStore,topic -v <DCStoreN>,"$SYS/Broker/<INName>/Monitoring/#"

2. DataCaptureStore

mqsicreateconfigurableservice <INName> -c DataCaptureStore -o <DCStoreN> -n backoutQueue,commitCount,commitIntervalSecs,dataSourceName,egForRecord,egForView,queueName,schema,threadPoolSize,useCoordinatedTransaction  -v "SYSTEM.BROKER.DC.BACKOUT","10","5",<DSN>","<ISName>","<ISName>","SYSTEM.BROKER.DC.RECORD","<IIBSchema>","10","false"

3. DataDestination

mqsichangeproperties<INName> -c DataDestination -o <DDName> -n egForReplay,endpoint,endpointType  -v "<ISName>","wmq:/msg/queue/<ReplayQName>@<IIBQMgrName>","WMQDestination"

Database connection configuration

1. Create odbc connection in ODBC Data Source Administrator (Demo =  <DSN>)
2. Set security connection information

mqsisetdbparms <INName> -n <DSN> -u <DBUsr> -p <BDPwd>

3. Run script DataCaptureSchema.sql from db2 command line (non-administrator). This script are available under
<IIBInstallation>\ddl\db2

Message Tree Copying Part II

The Compute node contains the following code:

DECLARE rVersion REFERENCE TO InputRoot.XMLNSC.ns:Inventory.ns:header.ns:version;
PROPAGATE TO LABEL(rVersion);
RETURN FALSE;

Message tree copying summary

This section contained a large amount of information relating to message tree copying, therefore this summary might be useful:

Table 1.

Don't change a message tree and copy it	If a message tree is ever changed, then the bitstream copy cannot take place, and extra parsing is needed.
Always copy at the parser folder level for the bitstream transfer to take place	When a message tree is copied, the high-speed bitstream transfer takes place only at the parser folder level.
Parsers must be created when not copying to OutputRoot	Only the OutputRoot correlation name in ESQL has the special property that a parser is created for parser name folders. Not copying to OutputRoot means that the bitstream copy cannot take place, and the whole of the source tree is parsed and then copied to the Environment tree.
Avoid overwriting fields when a tree is copied	When a message tree is copied, any existing children of target message tree field are detached first. Detaching fields is not the same as deleting fields, so they are still in scope, but are no longer attached to the tree.
Setting a field to NULL is NOT the same as deleting it	When you attempt to improve the memory usage of message tree usage, setting a field to NULL should never be used.
XML, XMLNS, and XMLNSC are not the same domains	The efficient bitstream transfer does not take place when the tree is copied between these domains. Both XML and XMLNS are deprecated.

• Don't change a message tree and copy it:
• Always copy at the parser folder level for the bitstream transfer to take place:
• Parsers must be created when not copying to OutputRoot
• Avoid overwriting fields when a tree is copied
• Forcing a copy of the message tree fields when required
• Setting a field to NULL is NOT the same as deleting it
• XML, XMLNS, and XMLNSC are not the same domains
• Avoid tree copying