Sunday, June 30, 2013

How to rollback/commit processing messages from WSO2 ESB when enabled JMS transaction

This blog post describes the below use-case where WSO2 ESB 4.6.0 act as the intermediate channel in-between two JMS queues.

The Use-case

  1. First messages will be de-queue to a proxy service deployed in WSO2 ESB from an queue in ActiveMQ message broker.
  2. Then inside ESB,the incoming messages will undergo with some message mediation flow and then the message will be send to another JMS queue deployed in JBoss server.
  3. Inside the ESB,following failures could be happen; 
          a) Failures during message mediation process failure inside ESB [for example if you 
              use  dbreportmediator/lookup mediator inside mediation flow,some times failures can 
              happen due to database connection failures]
         b) Message sending failure,due to endpoint is unavailability.[For example,the JBoss 
             server in which the endpoint queue deployed is down]

The requirement is if any of above failures happen,the message should not process further and send to endpoint,rather it should rollback to the queue of JMS provider [ActiveMQ].If none of failures happen the message should send to back-end endpoint.

So how to achieve above requirement with JMS?

It can be achieve via enabling JMS transactions enabled.In this scenario,there will be no more transaction resources like databases,thus here it can be used JMS local transactions enabled.To enable it ,add below property in approapriate JMS transport receiver config of axis2.xml file of ESB [{ESB}/repositry/conf/axis2]

 <parameter name="transport.jms.SessionTransacted">true</parameter>

Additionally,there should be a way to control message rollbacking/commiting based on message mediation flow status.This can be achieved via setting the correct JMS acknowledgement mode.The correct acknowledgement mode for this scenario is 'Client Acknowledgement' mode.The different acknowledgement modes support by JMS are;


Auto mode: The messages sent  from ActiveMQ are automatically acknowledged. This is the simplest mode and expresses JMS's power by enabling once-only message delivery guarantee.If we set this in above scenario,the message will de-queue as the message reach to ESB from ActiveMQ without waiting until the message processed.
 
Duplicates okay mode:The messages sent  from ActiveMQ are automatically acknowledged just like auto mode.Under rare circumstances, the messages might be delivered more than once. This mode enables at-least-once message delivery guarantee. 
 
Client mode: The messages sent from ActiveMQ are not acknowledged automatically. The application must acknowledge the message receipt. In above use-case,ESB should acknowledge ActiveMQ.This mode gives the application (rather than the JMS provider) complete control over message acknowledgement, at the cost of increased code complexity.


In above scenario the JMS message sending to the back-end is handle with ESB[JMS consumer],not by ActiveMQ [JMS provider].And until the message processed and send to backend,it need to pause de-queing the message from ActiveMQ to ESB.Thus it need to avoid JMS auto acknowledgement mode in this use-case.And based on ESB side failures/success,the message from queue in ActiveMQ should rollback or commit.For that on each failure/success statuses in message flow of ESB,it should acknowledge ActiveMQ,either to rollback or commit.And ESB need to send acknowledgement messages to ActiveMQ on its message mediation flow success/failure.In other words,its like sending 'client acknowledgement' messages to ActiveMQ.
To set 'Client Acknowledgement' in ESB,set below property inapproapriate JMS transport receiver config of axis2.xml file of ESB [{ESB}/repositry/conf/axis2]

<parameter name="transport.jms.SessionAcknowledgement"locked="true">CLIENT_ACKNOWLEDGE</parameter>

Then  as an failure occurred in message mediation flow,we need to rollback the processing message back to ActiveMQ queue, the below property SET_ROLLBACK_ONLY has to be set to true in relevant fault handler.

<property name="SET_ROLLBACK_ONLY" value="true" scope="axis2"/>

Once set as above,the above a) failure instances can be managed.Now the question comes,is it possible to cover message rollbacking to ActiveMQ queue again when the above b) failure instances also with JMS transactions and JMS client acknowledgement mode.Answer is yes,it's possible to do so.If your backend endpoint is;

A http/https endpoint -Then you need to use 'callout mediator' to send the messages to backend.NOTE-If you use 'send mediator',you cannot achieve,JMS message rollbacking /commiting based on endpoint failure,due to http and https transports are asynchronous.

A JMS endpoint- You can use one from both 'callout' and 'send' mediator to send the messages to backend as JMS is a blocking transport.NOTE -If you are using the 'send' mediator,make sure you have remove below axis2 level property by your proxy service configuration.

<property name="ClientApiNonBlockingaction="remove" scope="axis2"/>
Axis2 will spawn a new thread to handle each outgoing message. To change this behavior,
remove this property from the message.A sample proxy service configuration,would be as below;
To learn about JMS transactions and acknowledgement modes,please refer;

Saturday, June 29, 2013

Setting up monitoring outgoing message from WSO2 API Manager with TCPMon

There will be some requirements that need to monitor the outgoing message from WSO2 API Manager to real back-end endpoint.For example monitor JWT token value as explained in here.
This blog-post describes how to add TCPMon monitoring in between WSO2 AM and real endpoint.In a diagram,below is how we going to setup TCPMon with WSO2 AM and endpoint.

In other words,it means that the messages from WSO2 APIM should go to real endpoint via TCPMon tool.
Say,real back-end endpoint is a web-service running in WSO2 Application Server[WSO2 AS] called 'echo'.
The web-service endpoint will be as http://10.200.1.30:9763/echo

When you create an API from WSO2 API Manager,you have to give the 'Production URL' of Add API page in APIPublisher app pointing to above 'echo' web service endpoint as below.



But,if you need to place TCPMon in-between WSO2 APIM and WSO2 AS ,that means you need to direct messages from WSO2 APIM to WSO2 AS via TCPMon tool.
Thus first start TCPMon.You can do this by executing tcpmon.sh script which can be found from WSO2 Products bin directories or by running TCPMon externally.\




Define a listen port for TCPMon [here 8085] and then define the listener target hostname and port values as the back-end endpoint hostname[10.200.1.30] and port values[9763].
Then edit the 'Production URL' of the created API from WSO2 API Manager by changing its hostname and port values to map with TCPMon running hostname[localhost] and its defined listen port [8085] value.Hence as per above set TCPMon,the production URL of the API would be;

http://localhost:8085/echo


Once done above changes,try invoking the created API.You will able to monitor the request and response flow in between APIManager and real endpoint [App Server].


Passing end-user details from client to real backend endpoint via JWT token

In real-world business system,WSO2 API Manager useful on exposing company APIs, in a secured and controlled manner with the features provided by APIManager as;
  • OAuth support [To secure API invocations]
  • Throttling support [To control API invocations]
  • Monitoring support [To track API usage]
More technically what happening is when a user sends a particular API request,it will goes to WSO2 APIManager node and from there,the request will route to the real implemented back-end endpoint of the particular API and get back the response and returned it to the API invoked user. There can be a use-case,that this back-end endpoint may expect the details of API invoked user as to pass those details to some internal company usage  as;
  • Additional authentication/authorization
  • Track usage data from an internal system.
So how to support above requirement from WSO2 AM.
There comes the use of JSON Web Token[JWT] implementation done inside WSO2 AM. JWT is a means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JavaScript Object Notation (JSON) object that is used as the payload of a JSON Web Signature (JWS) structure or as the plain-text of a JSON Web Encryption (JWE) structure, enabling the claims to be digitally signed or MACed and/or encrypted.For get more information on JWT tokens ,please refer this link.
With current JWT token implementation of APIManager,following details will be pass to backend endpoint.

  1. Subscriber Name
  2. Application Name
  3. Tier
  4. End-User Name
  5. API context
  6. Version
  7. Key Type [Production/Sandbox]
  8. User Type[Application Token/User Token]
Below blog content will guide you on how to enable JWT token passing support from APIManager.

Configure WSO2 APIManager to pass JWT token

  1. Enable JWT token generation by uncommenting below configuration and set its value as true from api-manager.xml.
       <EnableTokenGeneration>true</EnableTokenGeneration>

    2. Start the AM server,create,publish API and subscribe it from the APIStore.


    3. Invoke the API and capture the outgoing message from the API Manager.Here its required to 

      monitor transport level header values as JWT token will be pass to backend as a transport header.

      To capture the outgoing message,you can use;

  • By Tcpmon tool in-between APIManager node and backend endpoint.For more info,refer the blog.
  • By enabling  wire logs of APIManager -By updating below configurations into log4j.properties file which can be found from {AM_Home}/repository/conf.If you follow this,you need to restart AM node.
        Uncomment  log4j.category.org.apache.synapse.transport.nhttp.wire=DEBUG
        Update log4j.category.org.apache.synapse.transport=INFO as  
        log4j.category.org.apache.synapse.transport=DEBUG

With capturing APIManager outputting message from one of either above option,you'll see a HTTP header  name as  "X-JWT-Assertion" has append to transport headers.Note the JWT-Assertion header value will contains base64 encoded value.If you capture the outgoing message of    APIM with wirelogs,when you invoke an API,you'll see below logs will be output in APIM with JWT-Assertion.

TID: [0] [AM] [2013-06-29 09:05:05,548] DEBUG {org.apache.synapse.transport.http.headers} -  http-outgoing-1>>X-JWT-Assertion: eyJ0eXAiOiJKV1QiLCJhbGciOiJTSEEyNTZ3aXRoUlNBIiwieDV0IjoiTm1KbU9HVXhNelpsWWpNMlpEUmhOVFpsWVRBMVl6ZGhaVFJpT1dFME5XSTJNMkptT1RjMVpBPT0ifQ==.eyJpc3MiOiJ3c28yLm9yZy9wcm9kdWN0cy9hbSIsImV4cCI6MTM3MjUyMjgwNTE3NywiaHR0cDovL3dzbzIub3JnL2NsYWltcy9zdWJzY3JpYmVyIjoibGFsYWppIiwiaHR0cDovL3dzbzIub3JnL2NsYWltcy9hcHBsaWNhdGlvbmlkIjoiMSIsImh0dHA6Ly93c28yLm9yZy9jbGFpbXMvYXBwbGljYXRpb25uYW1lIjoiRGVmYXVsdEFwcGxpY2F0aW9uIiwiaHR0cDovL3dzbzIub3JnL2NsYWltcy9hcHBsaWNhdGlvbnRpZXIiOiJVbmxpbWl0ZWQiLCJodHRwOi8vd3NvMi5vcmcvY2xhaW1zL2FwaWNvbnRleHQiOiIvYXBpMSIsImh0dHA6Ly93c28yLm9yZy9jbGFpbXMvdmVyc2lvbiI6IjEuMi4zIiwiaHR0cDovL3dzbzIub3JnL2NsYWltcy90aWVyIjoiVW5saW1pdGVkIiwiaHR0cDovL3dzbzIub3JnL2NsYWltcy9rZXl0eXBlIjoiUFJPRFVDVElPTiIsImh0dHA6Ly93c28yLm9yZy9jbGFpbXMvdXNlcnR5cGUiOiJBUFBMSUNBVElPTiIsImh0dHA6Ly93c28yLm9yZy9jbGFpbXMvZW5kdXNlciI6ImxhbGFqaSIsImh0dHA6Ly93c28yLm9yZy9jbGFpbXMvZW5kdXNlclRlbmFudElkIjoiLTEyMzQifQ==.YtVaDtRYtfUkVDvwe9V8oqsXK8OkB4HUhsQS2z3ngWRNjAktSKWlH+Is9T5EQnsg8hrsJQ4nKDdwDWHAUIFxIsb7bX/Y1O+WSLMLZYQ11WVzFaw50BJuqPbL9ZOfux1iRnm4ZbxClVSan72g/w8a05UnCvsGyIh5oCP4RUsAhXo= {org.apache.synapse.transport.http.headers}

To get the end user details,above base64 encoded string need to decode back.NOTE -Above encoded string contains three seperate encoded values.Those three are seperated from '.' [dot] sign. I have highlighted those three strings with different colors in above. You need to decode first two encoded strings to retrieve whole token structure.To decode,you can use www.base64decode.org link.
Once you decode,above first encoded string [green colored] ,you'll get an output as;

{"typ":"JWT","alg":"SHA256withRSA","x5t":"NmJmOGUxMzZlYjM2ZDRhNTZlYTA1YzdhZTRiOWE0NWI2M2JmOTc1ZA=="}
Above will output the token type as JWT and the algorithm used to encode the value as SHA256withRSA.



Then you can decode above second encoded string[red colored],you'll get end-user details as below.

{"iss":"wso2.org/products/am","exp":1372522805177,
"http://wso2.org/claims/subscriber":"lalaji",
"http://wso2.org/claims/applicationid":"1",
 "http://wso2.org/claims/applicationname":"DefaultApplication",
 "http://wso2.org/claims/applicationtier":"Unlimited",
 "http://wso2.org/claims/apicontext":"/api1",
 "http://wso2.org/claims/version":"1.2.3",
"http://wso2.org/claims/tier":"Unlimited",
"http://wso2.org/claims/keytype":"PRODUCTION",
"http://wso2.org/claims/usertype":"APPLICATION",
 "http://wso2.org/claims/enduser":"lalaji",
"http://wso2.org/claims/enduserTenantId":"-1234"}

Such that the overall Token Structure would be like below format;
{"typ":"JWT","alg":"SHA256withRSA","x5t":"NmJmOGUxMzZlYjM2ZDRhNTZlYTA1YzdhZTRiOWE0NWI2M2JmOTc1ZA=="}


{"iss":"wso2.org/products/am","exp":1372522805177,
"http://wso2.org/claims/subscriber":"lalaji",
"http://wso2.org/claims/applicationid":"1",
 "http://wso2.org/claims/applicationname":"DefaultApplication",
 "http://wso2.org/claims/applicationtier":"Unlimited",
 "http://wso2.org/claims/apicontext":"/api1",
 "http://wso2.org/claims/version":"1.2.3",
"http://wso2.org/claims/tier":"Unlimited",
"http://wso2.org/claims/keytype":"PRODUCTION",
"http://wso2.org/claims/usertype":"APPLICATION",
 "http://wso2.org/claims/enduser":"lalaji",
"http://wso2.org/claims/enduserTenantId":"-1234"}


How to pass Access Token used to invoke API to backend


With above JWT token approach,it'll pass the end user details and the invoked API data.But it will not pass the access token to backend. In WSO2 APIManager default behaviour,once an API request comes with an access token include in the Authorization header,it'll first processed the API Key validation handler and drop that access token contained Authorization header from message,without passing it to back-end endpoint.
If you want to pass an OAuth access token to back-end,modify the below configuration in api-manager.xml.

<RemoveOAuthHeadersFromOutMessage>false</RemoveOAuthHeadersFromOutMessage>


Thursday, June 6, 2013

Message Format Transformation from SOAP to POX

When WSO2 ESB used to process and forward some messages to a back-end endpoint,there will be chances,that back-end endpoint is expecting a XML message without a SOAP request.But at default,the messages which are pass through WSO2 ESB will be in SOAP format.
For example,say from WSO2 ESB ,we are passing messages to a queue/topic inside a JMS broker as ActiveMQ/JBoss. And the further processing on this queue by the backend will be expecting a XML message to be enqueue to the queue,instead a SOAP envelope.
Thus before send the message to JMS endpoint from WSO2 ESB,we need to convert the message format from SOAP to XML.
This can be done through modifying the WSO2 ESB endpoint element.
Add the attribute 'format=pox' to element of the configuration in the related WSO2 ESB proxy/sequence configuration as shown in below.


<endpoint name="jms-service-epr"><address format="pox"
uri="jms:/SimpleStockQuoteService?transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory&                 java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&java.naming.provider.url=tcp://localhost:61616" /
>