Thursday, February 24, 2011

REST API for message level and system level metrics for the XML Gateway

In my first few weeks at Vordel, I've been spending a lot of time on the "cool" part of the product - building circuits and policies. As a developer, I really enjoy (am amazed) at how quickly you can know things out. But this is a job after all, so I also have to spend time coming up to speed on some of the less "glamorous" yet really important and valuable features of the product.

I've started to look at the various administrative interfaces in the product and see what makes them tick. One really powerful feature of the Vordel XML Gateway is its out-of-the-box real time monitoring capabilities. The application itself is a Flash app that displays all of the details of the usage of the gateway - successful messages, blocked messages, failed messages - traffic to various remote hosts.

One thing that it also has is the system utilization - memory and CPU. This is really nice because you can get both the message (application level) and system-level information from a single source. This makes the task of integrating this information into an enterprise operations system much simpler. But how do you get at the information. Simple, http://gateway:8090/metrics



Notice that I'm just accessing the URL form a browser and you can see both the system information, as well as all of the statistics for the messages.

This is the same URL that the flash application accesses to retrieve the information in the real time monitor. The URL is protected with HTTP Basic authentication (all of the admin services are out of the box) so access is restricted as well as audited. Now that you know that the REST API is available, you can integrate this information into your environment.

Tuesday, February 22, 2011

How to Use an XML Gateway with an Asynchronous Web Service using WS-Addressing


In general synchronous web-services are simpler and more common than asynchronous web services. I like them, because for 99% of cases, the security can be done at the transport level using 2-way SSL. Asynchronous web-services introduce additional security challenges - mainly that messages are likely to be in memory or on disk where the transport is not there to keep the contents of the message secure. The purpose of this post is not to explore the security challenges of using asynchronous web-services, but another complexity - proper handling of web-services callbacks through an intermediary.

One of the main uses of an XML gateway is to encapsulate the end-point of the actual service from the caller. This approach is aligned with SOA best practices, but from a security perspective not letting people know where your service actual lives is a really good idea. This principal can also apply in the callback use case - we may want to hide the location of callback URL from the actual service - let's for the sake of this discussion consider it need to know. The end service can just callback to the gateway, and the gateway will deliver the message back to its appropriate destination.

Now assuming that we're not passing the location of the callback - in WS-Addressing this is called the wsa:ReplyTo address - then when the gateway finally receives the response, how does it know where to send the message? To solve this problem, the gateway needs to create a cache of replyTo URLs keyed off of the messageId. Each request has a wsa:MessageId. When the server sends its reply, the original message id is reference in the wsa:RelatesTo header. This way the gateway can correlate the request with the response.

Understanding the Example
I've created an example project that demonstrates how the gateway works in the above scenario. I'm using the gateway to play the role of 3 different use case actors - client, gateway, and server.


  • Client - SOAPBox is sending the initial request to the service. I created an HTTP service listening on port 11000 to serve as the destination of the final wsa:ReplyTo.
  • Gateway - When the service gets invoked, the gateway stores the wsa:ReplyTo in a cache (keyed by the wsa:MessageId) and modifies the wsa:ReplyTo and wsa:To fields accordingly. In the callback service, the gateway retrieves the original wsa:ReplyTo from the cache (pulling the key from the wsa:RelatedTo) field, and sends the request to the original wsa:ReplyTo
  • Server - This is just using the gateway to simulate the back end server. When it receives the message, it simply sticks the file on disk. The gateway has a directory scanner configured. Directory Scanner is the ability for the gateway to process files sitting on the file system. When the Directory Scanner finds a file, it modifies the response (read: appends "Hello" to the message) and then sends it to the address in the wsa:ReplyTo which is the gateway.
You can see what an end-to-end flow looks like in the Real Time Monitor

The Gateway Policy Details
The example contains policies for all three roles.


I wanted to spend some time here to review in a little more detail the gateway and what it does when it processes the request, and handles the callback. When the request arrives, the original message looks something like this:


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
<soap:Header>
<wsa:MessageID>uuid:6B29FC40-CA47-1067-B31D-00DD010662DA</wsa:MessageID>
<wsa:ReplyTo>
<wsa:Address>http://localhost:11000/callback</wsa:Address>
</wsa:ReplyTo>
<wsa:To>http://localhost:12000/SoapContext/SoapGreeterPort</wsa:To>
<wsa:Action>Greet</wsa:Action>
</soap:Header>
<soap:Body>
<x1:greetMeOneWay xmlns:x1="http://apache.org/hello_world_soap_http/types">
<!-- Element must appear exactly once -->
<x1:requestType>Asynch Client using WS-Addressing</x1:requestType>
</x1:greetMeOneWay>
</soap:Body>
</soap:Envelope>


The message gets picked up and processed by the following policy:



I'm using policy shortcuts to encapsulate all of the details, but you get the basic idea. The message leaving the gateway has been transformed to have the gateway address in the wsa:ReplyTo and the server's address in the wsa:To. This is the resulting message sent to the server.


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
<soap:Header>
<wsa:MessageID>uuid:6B29FC40-CA47-1067-B31D-00DD010662DA</wsa:MessageID>


<wsa:Action>Greet</wsa:Action>
<wsa:ReplyTo>
<wsa:Address>http://localhost:12000/SoapContext/GreetCallbackSoapPort</wsa:Address>
</wsa:ReplyTo><wsa:To>http://localhost:13000/SoapContext/SoapGreeterPort</wsa:To></soap:Header>
<soap:Body>
<x1:greetMeOneWay xmlns:x1="http://apache.org/hello_world_soap_http/types">
<!-- Element must appear exactly once -->
<x1:requestType>Asynch Client using WS-Addressing</x1:requestType>
</x1:greetMeOneWay>
</soap:Body>
</soap:Envelope>


So when the reply comes back from the server, and arrives back at the gateway, it has a new wsa:messageId, but the original messageId is available in the wsa:RelatesTo header.


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>
<wsa:Action
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" />
<wsa:MessageID
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
uuid:Id-0000012e5303f409-0000000000a96fd2-2
</wsa:MessageID>
<wsa:RelatesTo
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
uuid:6B29FC40-CA47-1067-B31D-00DD010662DA
</wsa:RelatesTo>
<wsa:To xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
http://localhost:12000/SoapContext/GreetCallbackSoapPort
</wsa:To>
</soap:Header>
<soap:Body>
<x1:greetMeResponse
xmlns:x1="http://apache.org/hello_world_soap_http/types">
<!-- Element must appear exactly once -->
<x1:responseType>
Hello Asynch Client using WS-Addressing
</x1:responseType>
</x1:greetMeResponse>
</soap:Body>
</soap:Envelope>


The callback policy then picks this message up and applies the following filters:




This is the basic caching pattern used by the gateway. The cache's key is the wsa:messageId set by the incoming request. This can then be retrieved from the cache by pulling the wsa:RealtesTo messageId. The resulting URL is then set as the destination (wsa:To) of the response, and the reply is sent. The final message looks like this:


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Header>
<wsa:Action
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" />
<wsa:MessageID
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
uuid:Id-0000012e53675a57-0000000001f76db2-18
</wsa:MessageID>
<wsa:RelatesTo
xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
uuid:6B29FC40-CA47-1067-B31D-00DD010662DA
</wsa:RelatesTo>
<wsa:To xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing">
http://localhost:11000/callback
</wsa:To>
</soap:Header>
<soap:Body>
<x1:greetMeResponse
xmlns:x1="http://apache.org/hello_world_soap_http/types">
<!-- Element must appear exactly once -->
<x1:responseType>
Hello Asynch Client using WS-Addressing
</x1:responseType>
</x1:greetMeResponse>
</soap:Body>
</soap:Envelope>
I've exported this project and added it to the Vordel Incubator. You can download the config here. Let me know what you think.















Friday, February 18, 2011

Architectural Pattern for XML Gateway - Service Bus Sandwich


I was in a discussion yesterday with a customer and the question of "Do I need an internal gateway in addition to my ESB?" came up. In general, people thing about XML gateways as the security appliance that you stick in the DMZ to prevent unauthorized access to services (i.e. the Super PEP). So, why do I need another one in my internal network?

Service Bus Sandwich

I think the most basic reason is that the system that does security processing at the front of the bus should be the same system that does security processing at the back of the bus. This promotes a consistent approach - same policies, same certificates, same administrative interfaces. This also optimizes the capabilities of each component in the architecture - XML Gateway off loads the security processing from the ESB - SSL Termination, XML threat checking, authentication, authorization, message encryption/signing etc. - and the ESB does what it does best - routing, transformation, service virtualization, protocol translation etc.

So, with all of the security processing off loaded to the XML Gateway, how should the ESB and the XML Gateway work together to pass security context. I think the simplest way is to simply use 2-way SSL between the front XML Gateway and the 2-way SSL between the back XML Gateway. This ensures that the ESB is receiving valid messages from the XML Gateway and that it can continue processing the message. Likewise for the back XML Gateway - since it trusts that the request is from the ESB, it can send the message off to its next destination. This makes the sole security interface between the XML Gateways and the ESBs a common trusted Certificate Authority.

If more information needs to be passed form the XML Gateway to the ESB - like the identity of the calling service/user, then you can simply use SAML bearer profile. This profile basically means "trust this assertion if you want.". Since we know the request is over 2-way SSL, bearer seems fine. You could optionally have the XML Gateway or STS sign the assertion, but its probably not necessary. If you're inclined to add the additional control of the signed assertion, you're only adding a public key/CA to the ESB - still a pretty minimal binding.

Looking at the back XML gateway, this is basically the Cloud Service Broker - mediating out access to services in the cloud - but what if you're not using any cloud services yet. Do I still need have the 2nd XML Gateway? Besides the benefits of centralization and consistency, lets look at another real world use case. Information arriving at the front gateway was secured in transmission using 2-way SSL, but contains PII that needs to be masked from the ESB while being processed - it can't be exposed in the clear, in logs, etc. - but the target service requires that information for processing. In this case the front XML Gateway can encrypt/mask the contents of the message and the back gateway can decrypt/unmask the message prior to being sent to the back end service.

In summary, in architectures that contain an ESB, having both a front XML Gateway to secure incoming traffic and a back XML Gateway to secure the outbound traffic provides real benefit. This may or may not represent two physically separate deployments - it just depends if the out bound traffic is allowed to be sent back out through the DMZ. The trust among the components can be done via transport - typically 2-way SSL. This greatly simplifies the administration of security policies, and optimizes the utilization of both the ESB and the XML Gateway.

Tuesday, February 15, 2011

It Takes A Village....Announcing the Vordel Incubator


After much debate, I'm pleased to announce the Vordel Incubator. I'm modelling after the wildly successful Oracle Coherence Incubator. The idea is to work as a community - company, customers, partners - on building solutions to real-world problems.

I've started the project off very modestly with a publishing of the full Maven project that I discussed earlier. As people know, I'm very interested in Maven and believe it is a really good tool for integrating a product into an enterprise SDLC. We'll be using it for the incubator. As such, there is some work to do in short order to extend the existing Maven support through out the fuller lifecycle - automated testing, deployment etc. I eager to work on these topics.

Additionally, as my team works with customers, there a common requests for solutions or examples that everyone would benefit from. Its my intention to grow a library of such solutions in the incubator as well.

Ultimately what I want is to try to harness the enthusiasm of the customers/partners that I've met over my first 6 weeks at Vordel - and channel into building things together that benefit everyone. But to do that, I need people's help and participation. If you are interested in participating in any way, please let me or anyone at Vordel know, and we'll get you looped into what we're doing. I've created a public google group to host discussions, so you can express your views/interests there.

I look forward to working with all of you on this.

Monday, February 14, 2011

How to extend the Vordel XML Gateway with Maven

One of the strengths of the Vordel XML Gateway is the ability of the product to be extended using Java. You can do this in two ways: JavaScript and through writing custom Filters. The really clever part about how the XML Gateway is engineered is that the underlying XML layers are written in native code so the product is really fast even when you have to use Java to customize it.

Recently, I had a customer ask "How do I build filters using Maven?". This is obviously a person after my own heart. I had spent a lot of time last year working with Maven, in particular for the OES/Spring/JBOSS/AOP integration I did. So, these are the steps for building the Example Filter using Maven.

Step 1 - Load the Dependencies into the Local File System


This for me is always the trickiest part. For things that are "non-Maven" you need to get them loaded as dependencies - but how? In order to get the XML Gateway APIs loaded into the local repository, you need to use the install:install-file goal to load some of the jars. As I continue to invest in this solution, look for other more elegant approaches, but for now this is what I'm going with.

This is the commands I ran, in my linux environment:


mvn install:install-file -Dfile=/opt/vordel/vordelgateway/system/lib/circuit.jar -DgroupId=com.vordel.vordelgateway -DartifactId=circuit -Dversion=6.0.3 -Dpackaging=jar

mvn install:install-file -Dfile=/opt/vordel/vordelgateway/system/lib/server.jar -DgroupId=com.vordel.vordelgateway -DartifactId=server -Dversion=6.0.3 -Dpackaging=jar

mvn install:install-file -Dfile=/opt/vordel/vordelgateway/system/lib/entityStore.jar -DgroupId=com.vordel.vordelgateway -DartifactId=entityStore -Dversion=6.0.3 -Dpackaging=jar

mvn install:install-file -Dfile=/opt/vordel/vordelgateway/system/lib/manager.jar
-DgroupId=com.vordel.vordelgateway -DartifactId=manager -Dversion=6.0.3 -Dpackaging=jar

mvn install:install-file -Dfile=/opt/vordel/vordelgateway/system/lib/common.jar
-DgroupId=com.vordel.vordelgateway -DartifactId=common -Dversion=6.0.3 -Dpackaging=jar

mvn install:install-file -Dfile=/opt/vordel/vordelgateway/system/lib/client.jar
-DgroupId=com.vordel.vordelgateway -DartifactId=client -Dversion=6.0.3 -Dpackaging=jar

mvn install:install-file -Dfile=/opt/vordel/policystudio/plugins/org.eclipse.gef_3.2.101.v20070814.jar -DartifactId=gef -DgroupId=org.eclipse -Dversion=3.2.101 -Dpackaging=jar

mvn install:install-file -Dfile=/opt/vordel/policystudio/plugins/org.eclipse.jface_3.3.1.M20070910-0800b.jar -DgroupId=org.eclipse -DartifactId=jface -Dversion=3.3.1 -Dpackaging=jar

mvn install:install-file -Dfile=/opt/vordel/policystudio/plugins/org.eclipse.swt.gtk.linux.x86_3.3.2.v3347.jar -DgroupId=org.eclipse -DartifactId=swt -Dversion=3.3.2 -Dpackaging=jar


Step 2 - Create a Parent POM


The parent POM references all of the dependencies, as well as simplifies the rest of the building process:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.vordel.vordelgateway</groupId>
<artifactId>parent</artifactId>
<version>6.0.3</version>
<packaging>pom</packaging>
<build>
<plugins>
<plugin>
<version>2.3.1</version>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>
</archive>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>com.vordel.vordelgateway</groupId>
<artifactId>circuit</artifactId>
<version>6.0.3</version>
</dependency>
<dependency>
<groupId>com.vordel.vordelgateway</groupId>
<artifactId>server</artifactId>
<version>6.0.3</version>
</dependency>
<dependency>
<groupId>com.vordel.vordelgateway</groupId>
<artifactId>entityStore</artifactId>
<version>6.0.3</version>
</dependency>
<dependency>
<groupId>com.vordel.vordelgateway</groupId>
<artifactId>manager</artifactId>
<version>6.0.3</version>
</dependency>
<dependency>
<groupId>com.vordel.vordelgateway</groupId>
<artifactId>client</artifactId>
<version>6.0.3</version>
</dependency>
<dependency>
<groupId>com.vordel.vordelgateway</groupId>
<artifactId>common</artifactId>
<version>6.0.3</version>
</dependency>
<dependency>
<groupId>org.eclipse</groupId>
<artifactId>jface</artifactId>
<version>3.3.1</version>
</dependency>
<dependency>
<groupId>org.eclipse</groupId>
<artifactId>gef</artifactId>
<version>3.2.101</version>
</dependency>
<dependency>
<groupId>org.eclipse</groupId>
<artifactId>swt</artifactId>
<version>3.3.2</version>
</dependency>
</dependencies>
</project>


With the POM created, run mvn install to load the POM into your local repository

Step 3 - Create the Example Filter Project


I used eclipse and the Eclipse Maven Plugin. I created a new Maven project that referenced the Parent POM


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>parent</artifactId>
<groupId>com.vordel.vordelgateway</groupId>
<version>6.0.3</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>com.vordel.vordelgateway</groupId>
<artifactId>example-filter</artifactId>
<version>0.0.1-SNAPSHOT</version>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.0.2</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
</plugins>
</build>
</project>


I then moved some files from the original example to conform with the Maven structure. I moved the simple.gif and resource files to their proper Maven place under resources. I also moved the MANIFEST.MF file to src/main/resources/META-INF so it is included in the jar.



You should be able to build the project. All that's left to do is copy the jar to the /opt/vordel/vordelgateway/ext/lib and the /opt/policystudio/plugins directories, and your plugin should be good to go.

Friday, February 11, 2011

How to retrieve an OAuth token from a WS-Trust based Security Token Service (STS)

I'm finally back home after 4 straights days in airports. During the week I delivered a really interesting use case that I wanted to share. This was in support of a demo where the customer wanted to understand how OAuth works with the XML Gateway. Given the natures of POCs, I had already built much of the demo around the customer's other requirement - retrieving a SAML assertion via a WS-Trust based STS. I had to come up with a way to add the OAuth functionality to the existing scenario. I think the approach that I came up with is novel and so I wanted to share it on the blog.

If you look at a WS-Trust RST (Request for Security Token) message, the initiator can request a token type, so the call to the STS can reasonably start with a request for an OAuth token, but the real question is "What to return and how to return it?". One thing that I liked about this particular scenario was that it gave me a change to dig a little more deeply into OAuth 1.0. I think I understand it much better. There have been many places where OAuth has been explained, so I won't cover everything, but basically for the Gateway scenario calling a service protected with OAuth, you need an access token and a corresponding access token secret. For the purposes of this scenario, I assumed that the access token and access token secret were already available to the STS - let's just say this happens when the user elects to allow the gateway application access to its information.

So, then how should we return this information. Both the access token and access token secret are just strings. The solution I used was to return them as Attribute Statements inside of a SAML assertion. One of the great things about SAML is its ability to include attributes related to the subject. The eliminated the need to come up with a custom token or simply pass them pack in the transport. The transport is OK, but I like the elegance of using SAML and quite frankly how easy it was to implement using the gateway.

On the caller side, I needed to modify the request to the STS to have the right token type
and then retrieve the assertion from the response.


The XML Gateway has a single filter that retrieves attribute assertions...couldn't have been easier.
On the STS side, this was also really simple. I just added a branch in the STS policy to handle the case where an OAUTH token has requested and the call the policy to add the SAML Attribute Assertion.
A quick re-deploy of the policy and I was in business. For the purposes of this demo, I used the LinkedIn API which is protected by OAuth, and made a simple call to get my network updates for the day.

I did all of this in about 1 hour very late at night - another strong testimony for the tool. The fact that I could get and set the SAML assertions so easily, really made the whole thing "just work".

I know that OAuth 2.0 has a binding for passing a SAML assertion to retrieve an access token, but the convergence between SAML and OAuth is definitely underspecified.
The scenario I'm showing here is the binding of an OAuth access token to a SAML assertion. From a specification perspective, I think all you would need to formalize are the constants for requesting the token from the STS (token type) and the attribute names (OAuth access token) themselves. I'll get Mark on it after he gets back from RSA :)

Tuesday, February 1, 2011

How to Secure the Cloud?


I'm in the process of getting the Boston chapter of the Cloud Security Alliance started. I'm just waiting for the "paperwork" to go through, but I'm really excited about what I'm hearing from customers about the cloud. Coming from Oracle, you get a bit of the "Larry Hates the Cloud" mindset, but in my limited time here at Vordel, I can see the deep interest from customers.

Mark O'Neill has published a few articles recently on a few topics within cloud security (SSO to Google Mail and Security Checklist for Cloud Security) but there is single "Cloud Security" solution. Probably the only term less well defined than "Cloud" is "Security". CSA is starting a whole new focus are on "Security as a Service" - again we could have/and will continue to have a debate over what is a "Service".

Unlike SOA, IT people are being asked by the business "What are we doing about the cloud?". This is because the "cloud" model continues to drive real cost savings. So, given the state of security in the cloud....what should you do?


From my perspective, the technical alignment between the traditional XML Gateway use cases and the Cloud Service Broker use cases suggests that though the space and the standards are evolving, an investment in an XML Gateway will provide significant value moving up into the cloud. What do you think?