How can we use existing infrastructure to define a flow of messages between services?

The problem

I have been asked this by many Grid folks in the context of workflow execution. There are many situations where one, using a workflow language (the e-Science community has many of these), wants to describe a flow of messages between services without those messages ever coming back to a coordinator (the workflow enactor). For example, the result of a query may have to flow from one service to another for processing. If that result is big, we don't want to receive it back only to send it to the next service in the (work/data)flow. Yes, I know... a classical problem with many existing solutions.

This came up as part of my recent discussion with Paul. The problem is actually the generalisation of the one I described in my "third-party delivery" post. There, I used the ReplyTo information header from WS-Addressing to request from the receiving service that the reply message be delivered to a third party. I used WSE 2.0 and C# to demonstrate how this could be done.

As I said, there have been many solutions to this problem over the last few decades (graph execution for functional/dataflow languages, distributed workflow enactors, etc.). But I wanted to know whether a simple solution could be found using existing WS technologies.

Some solutions

One could suggest that this is an application-level requirement and that it should be dealt by the service logic. In such a solution, the description of the required flow is carried in the messages' payloads and the service logic at each receiving service takes the decision on where to send its "reply" message. Each service's message exchange patterns have to be designed to accept this extra information. Consider the following message:

<soap:Envelope>
<soap:Header>
...
</soap:Header>
<soap:Body>
<myns:data>
...
<flow:route>
<epr>EndpointReferenceForService1</epr>
<epr>EndpointReferenceForService2</epr>
...
</flow:route>
</myns:data>
</soap:Body>
</messaging>

Each service that receives the above message must remove its endpoint reference (EPR) from the flow and then pass the remaining EPRs to the next service. There are many issues with this approach. The service logic of all services will have to be changed to deal with flow of messages, which should really be dealt by the infrastructure.

One could just define a new protocol so that infrastructure at each service can process the flow-related information.

<soap:Envelope>
<soap:Header>
<flow:route>
<epr>EndpointReferenceForService1</epr>
<epr>EndpointReferenceForService2</epr>
...
</flow:route>
...
</soap:Header>
<soap:Body>
<myns:data>
...
</myns:data>
</soap:Body>
</messaging>

Now, the implementation of this new protocol would have to be deployed at each service although the service logic remains unaffected.

Could there be a way that doesn't require changes to service logic or new protocol specifications? What would the limitations be?

Using WS-Addressing

I set out to investigate whether existing infrastructure could be used. Assuming that WS-Addressing is a specification that everyone should support (but not everyone should support specifications that use WS-Addressing :-), what can we achieve?

As with the "third-party delivery" case, I use the ReplyTo information header to instruct the receiving service to send the "reply" message to a different service. I make the assumption of course that the services in the middle of the flow support a request-response message exchange pattern. Otherwise, there isn't a point in including them in the flow. The flow I am going to try to implement looks like this:

Initiating service -> Service 1 -> Service 2 -> ... -> Service N

The trick is to instruct the WS-Addressing supporting infrastructure to include in the "reply" message the necessary information headers. This is done through the use of the ReferenceProperties element in an endpoint reference (EPR) like the ReplyTo element. (In the new version of WS-Addressing, the ReferenceParameters element would be used instead)

<soap:Envelope>
<soap:Header>
<wsa:Action>...</wsa:Action>
<wsa:To>Service1</wsa:To>
<wsa:ReplyTo>
<wsa:Address>Service2</wsa:Address>
<wsa:ReferenceProperties>
<wsa:ReplyTo>
<wsa:Address>Service3</wsa:Address>
<wsa:ReferenceProperties>
<wsa:ReplyTo>...
</wsa:ReferenceProperties>
</wsa:ReplyTo>
</wsa:ReferenceProperties>
</wsa:ReplyTo>
<soap:Header>
<soap:Body>
<myns:data>
...
</myns:data>
</soap:Body>
</messaging>

Service 1 receives this message, processes the payload, and sends a message to Service 2 as instructed by the outer ReplyTo header. According to the WS-Addressing specification, the Address element will become a To header in the outgoing message. Also, the ReferenceProperties element must be included as a header in the outgoing message, which will look like this:

<soap:Envelope>
<soap:Header>
<wsa:Action>...</wsa:Action>
<wsa:To>Service2</wsa:To>
<wsa:ReplyTo>
<wsa:Address>Service3</wsa:Address>
<wsa:ReferenceProperties>
<wsa:ReplyTo>...
</wsa:ReferenceProperties>
</wsa:ReplyTo>
<soap:Header>
<soap:Body>
<myns:data>
...
</myns:data>
</soap:Body>
</messaging>

As the message from Service 2 arrives to Service 3, the same process takes place.

This is a great use of ReferenceProperties and one to which I do not object (I don't like the use of ReferenceProperties as an object/resource key in WS-RF but that's just me).

Advantages/Disadvantages

The advantage of this approach is that no new infrastructure or service logic is necessary. All that is required is support for WS-Addressing. The services in the flow don't really know that they are part of a flow. Also, there is no need for the initiator to orchestrate the flow of the messages at every step.

Of course, the last advantage can also be a disadvantage since the initiator of the flow doesn't have control of what is going on. A FaultTo EPR can be included so that in case of a fault someone gets notified. Finally, there can be no branches in the flow of messages as in traditional (control/data)flow systems.

Implementation

In order to test the validity of the above, I wrote a small test using C# and WSE 2.0 SP1. The code is very simple so allow me to take you through it.

The following is the implementation of a service. Each service in the flow is implemented in exactly the same way. You'll notice that the service only has to deal with the processing the data and its completely unaware of whether the service logic is executed as part of a flow or not.*

namespace uk.ac.neresc.multinodeflow.service2
{
public class Service : SoapService
{
[SoapMethod("service:process")]
public XmlElement Process(XmlElement data)
{
// Process data ResponseSoapContext.Current.Addressing.Action = "service:process"; return data;
}
}
}

You may also notice that I am cheating and I have to assign a value to the Action header inside the body of the service logic. There is a reason for this which is related to the WSE implementation and not the general idea. More on this later.

The code for the initiator service is straight forward (the URIs of the 3 services in the flow are received as command-line args):

namespace uk.ac.neresc.multinodeflow.consumer
{
class Program
{
static void Main(string[] args)
{
SoapEnvelope envelope = new SoapEnvelope();
envelope.Context.Addressing.Action = new Action("service:process"); // Body
envelope.CreateBody().InnerXml = "<message>hello world</message>"; // Info about the third service
envelope.Context.Addressing.ReplyTo = new ReplyTo(new Uri(args[1])); // Info about the forth service
System.Xml.XmlDocument doc = new System.Xml.XmlDocument();
envelope.Context.Addressing.ReplyTo.ReferenceProperties = new ReferenceProperties();
envelope.Context.Addressing.ReplyTo.ReferenceProperties.AnyElements.Add(
new ReplyTo(new Uri(args[2])).GetXml(doc)); SoapSender sender = new SoapSender(new Uri(args[0])); sender.Send(envelope);
}
}
}

WSE and the Action header

I shouldn't have to change the Action header in each service because that's an infrastructure matter. The reason I have to do this is because of how WSE uses the ActionWS-Addressing header. In a request-response message exchange pattern, WSE will append the word "Response" to the value of the Action header. Since WSE uses the Action header for dispatching purposes (based on its value the appropriate method of the derived SoapService class is called) the following service receiving the message will not be able to dispatch its service logic (note that all services have a method with the same "service:process" action).

Another solution to this problem would be to implement the service logic using the SoapReceiver class and overload the Receive() method. The dispatching of Receive() does not require the Action header but the implementation would have to deal with the ReplyTo headers directly.

And while I am on the subjection of the Action header... Why is it mandatory? Why is "Response" appended at the end of its value? It can't be for correlation purposes since the RelatesTo header could/should be used for that. Why do we need Action at all anyway? Shouldn't the dispatching of some service logic depend just on the payload? I think Jim was going to write something about this.

Conclusion

We had a look at how WS-Addressing could be used to describe a flow of messages between services in order to meet a common requirement in the Grid community. It's not the best solution but it has the advantage of not requiring changes to already deployed services, assuming that support for WS-Addressing exists. We also saw an example on how this could be implemented using WSE's support for WS-Addressing.

Any comments/corrections/disagreements/alternatives are more than welcome and appreciated!

* Enabling redirections

In order to make redirections (ReplyTo and FaultTo) work at the service side you'll need to add the following in your WSE's configuration:

<microsoft.web.services2>
<messaging>
<allowRedirectedResponses enabled="true" />
</messaging>
</microsoft.web.services2>