Part 1 of this two-part post briefly introduced Dynasoar and how Virtual Machines could be used as the unit for service deployment. As Tim Freeman correctly pointed out in his comment, there is plenty of VM-related work out there. I didn't want to suggest that our work on VMs was unique; not at all. We are just investigating the use of VMs within the context of Dynasoar's architectural approach to service deployment and also attempt a quantitative analysis of the performance implications. We are very much aware of the great work done by other research groups in the VM field and, in fact, we have already started exchanging ideas with them.
In this second part, I am going to discuss my experience of building a Dynasoar prototype using Indigo (erm... Windows Communications Framework, WCF) and Virtual Server 2005.
The architecture
At the end of Part 1 I included a very high-level architecture diagram illustrating the major components of the system. The implementation closely follows that architecture. Here's that diagram again:
The Service Provider service
The Service Provider's role is to make services available to the network and hide the Dynasoar infrastructure from Consumers. The prototype implementation for the service is very simple. Upon receipt of a message it checks to see if the service logic has been deployed on a Host Provider:
- If it hasn't, then a message exchange to perform the deployment (of the virtual machine image) takes place.
- If it has, then the arriving message gets routed to the Host Provider where the service has been deployed.
The Savas.Dynasoar.ServiceProvider.Service class implements the IRequestReplyMessageIntermediaryWCF interface (for the purposes of the prototype I assumed request-response interactions between Consumer and service). For each Web Service we want to make available on the network, we register the Savas.Dynasoar.ServiceProvider.Service at an endpoint and make it available to Consumers. Currently, I use the InstanceMode.Singleton behaviour in order to maintain the service's state between message interactions. The endpoint of the deployed service at the Host Provider is maintained as part of that state. A better approach, and the one that the Dynasoar folks have implemented in Java, would have been to use a registry for persisting the address of the deployment. Here's some pseudocode:
[ServiceBehavior(InstanceMode = InstanceMode.Singleton, ValidateMustUnderstand = false)]
public class Service : IRequestReplyMessageIntermediary
{
private EndpointAddress serviceAddress = null
public Message ProcessMessage(Message message)
{
// Implement some locking and message queuing to deal
// with requests arriving while the deployment takes place
if (this.serviceAddress == null)
{
// We need to deploy the service
hostProviderProxy = // Create an MTOM binding and channel
// to the HostProvider for deployment
vm-package = // Create a deployment package with the VM image
this.serviceAddress = hostProviderProxy.Deploy(vm-package)
}
// Forward the message
hostProviderProxy = // Create a binding and channel to the
// this.serviceAddress endpoint
// for message processing
return proxy.ProcessMessage(message)
}
}
The Host Provider service
This part of Dynasoar is a little bit more complicated. First, some deployment-related decisions have to be made. When a virtual machine is received, we have to decide whether it would be directly accessible from the outside network (and, as a result, from the Service Providers) or whether it would be given an IP address from a private virtual network (Virtual Server 2005 supports virtual networks). The former option required a set of predefined MAC-IP pairs to be registered with my University's DHCP server. This was not an option so I had to go with the latter approach which meant that another indirection would have to be introduced; an WCF service was implemented which receives messages from Service Providers and then forwards them to the appropriate Virtual Machine. As it was the case with the Service Provider, the implementation of this part of the Host Provider implements the IRequestReplyMessageIntermediary interface (again, the assumption of request-response interactions is made). Here's the first part of the pseudocode for the Savas.Dynasoar.HostProvider.Service:
[ServiceBehavior(InstanceMode = InstanceMode.PerCall, ValidateMustUnderstand = false)]
public class Service : IRequestReplyMessageIntermediary
{
public Message ProcessMessage(Message message)
{
virtualMachineHeaders = // Get VM-specific headers from message
endpointAddress = // Use virtualMachineHeaders to construct an
// endpoint reference
proxy = // Create a binding and a channel to that endpoint
// Forward the message
return proxy.ProcessMessage(message)
}
}
Each message arriving from a Service Provider will contain Dynasoar-specific SOAP headers with information about the targeted virtual machine. This way, we don't have to maintain any state at the Host Provider. Each message carries all the information necessary for the message to be directed to the correct Virtual Machine. Here's an example of such a SOAP message.
<soap:Envelope>
<soap:Header>
<dynasoar:ServiceGuid>8249c86c04b2401c9264d6f8acdfa2a0</dynasoar:ServiceGuid>
<dynasoar:LocalEndpoint>http://10.246.0.17/service.svc</dynasoar:LocalEndpoint>
<wsa:To>http://11.0.0.10/HostProvider</wsa:To>
<!- ... ->
</soap:Header>
<soap:Body>
<!- message to the service ->
</soap:Body>
</soap:Envelope>
How did those Dynasoar SOAP headers get inserted into the message? As we saw earlier, the implementation of the Service Provider did not add any SOAP headers to the message; it's just forwarded it.
Here I used the Reference Parameters (ReferenceProperties in the current implementation of WCF) feature of WS-Addressing. When a Virtual Machine gets deployed, the returned endpoint reference contains as reference parameters the two Dynasoar-specific elements we see above. When the Service Provider creates a channel to that particular endpoint, WCF automatically adds the two elements as SOAP headers as per the WS-Addressing specification. (According to yesterday's W3C Candidate Recommendation of WS-Addressing an attribute would have to appear indicating that these headers got included into the message because they were Reference Parameters).
Sidenote: I think this is a good example of the intended use of Reference Parameters. The consumer of the endpoint reference (in this case the Service Provider) is completely unaware of the inclusion of headers. From an architectural point of view, the consumer still thinks that the interaction takes place with a service. This is in contrast to the way WSRF and WS-Transfer (PDF) use reference parameters to identify a specific resource behind the service boundaries, promoting an architectural view of the distributed application where interactions take place with resources rather than services.
So, here's some pseudocode for the part of a Host Provider that supports the deployment of VMs:
[ServiceContract(Namespace = ..., Name = ..., FormatMode = ...)]
public interface IHostProvider
{
[OperationContract(Name = ..., Action = ...)]
EndpointAddress Deploy(DeploymentOptions package);
}
[ServiceBehavior(InstanceMode = InstanceMode.PerCall, ValidateMustUnderstand = false)]
public class Service : IHostProvider
{
public EndpointAddress Deploy(DeploymentOptions package)
{
// Create a new GUID for the VM
// Get the VM image from the package and save it to disk
// Create a new Virtual Server VM and Network via COM
// Start the VM
// Create the endpoint reference with information about the VM
// and return it to the Service Provider
}
}
Very simple indeed. There are some security-related issues with .NET - Virtual Server COM interoperability that need to be considered but I will not discuss those in this post.
These are the implementations of the two Dynasoar services. Really trivial as you can see.
Transferring Virtual Machines
A Virtual Machine image is not the smallest file one can transfer around. In fact, my test environment uses a VM image of 1.6 GB. The image includes a Windows Server 2003 installation, IIS, the WCF runtime, and an 'Echo' service deployment. Here are some options for the deployment:
- Placing 1.6 GB in a SOAP message is probably not the best way to transfer the image, even when MTOM is used 🙂 Instead, I made the assumption that the base Virtual Machine image was already distributed to the Host Providers used. The base VM had Windows Server 2003, IIS, and WCF installed. Then, a differencing Virtual Hard Disk (VHD) was used to deploy an 'Echo' service. The base-VM assumption allowed me to only have to transfer that differencing hard disk. It turns out that the VHD was ~0.5GB even though the deployed service was just few KBs large (I think it's the virtual memory space that gets allocated by default when you boot the differencing disk). However, despite its large size, a VHD is highly compressible: 0.5GB gets gzipped to ~9MB. I can transfer 9MB as an MTOM attachment in one SOAP message (I had to change the transport's MaxMessageSize to do that of course). At the Host Provider, I use .NET 2.0's gunzip API to uncompress the differencing image.
- Rather than assuming a base VM which is already distributed, one could send a URL to the Host Provider for a Web-friendly way of transferring the image (e.g. an FTP URL). The Host Provider receives the URL and then fetches the VM. The Service Provider and Host Provider could also agree on using a fast transport (e.g. GridFTP).
- One could transfer the saved memory state of a suspended VM together with the differencing Virtual Hard Disk or the Virtual Machine image. Here are some reasons why we may want to do this: a) Increased start up time for the VM since the OS doesn't have to do a cold start, at the cost of extra communication cost; b) Maintaining the runtime environment of the VM at the state it was left by whoever configured it; c) Enable mobile VMs scenarios where a VM is suspended and transferred from one Host Provider to another. Deployments to enable scientific computing seem to benefit from such approaches.
Performance
As one would expect, the transfer of large VM images across the network significantly affects performance, at least during the deployment phase. Since I was doing request-response interactions from the Consumer to the Service Provider over HTTP, I had to increase the HTTP timeout in order to accommodate for the >90 secs it required to transfer, deploy, and start the VM (my laptop hosted all the Consumer, Service Provider, and Host Provider actors). The cost of redirecting a SOAP message from the Service Provider to the Host Provider and the Host Provider to the VM is not significant. However, I expect that WCF and Windows optimise communications on the same machine and, hence, once the tests are performed on multiple machines the experience may be different. That's why I leave the numbers for a different post or a future paper.
Things to measure include:
- The performance implications on using different deployment approaches (differencing disk vs VM image vs either with saved state);
- The performance implications of the redirection;
- The granularity of the service logic that makes the introduced cost of Dynasoar acceptable;
- The performance benefits of using alternative mechanisms for transfering the images (e.g. P2P, GridFTP, TCP/IP sockets, etc.).
- and others
Future work
The work on Dynasoar continues by the excellent team of people in Newcastle. Since I'll be moving to Microsoft, I won't be directly involved with it anymore although I am planning to do some more work on it and perhaps write a paper, if I can dedicate some free time while in the state of Washington.
We have already written a paper with our experience on extending the Service Providers with load-balancing capabilities based on information collected from Host Providers. The work was done by Fabio Scibilia who visited us for few months from Italy. His Service Usage Monitoring framework for Dynasoar lets the Service Providers collect and analyse QoS-related information from the Host Providers and, based on that information, decide on which of the available Host Providers to deploy a service or a VM.
The analysis of policies, service-level agreements, capability/requirement characteristics, semantics, and other metadata can be used by Service Providers when making deployment or message routing decisions. This could be another area for future work, which of course relates to the MEST ideas of declaratively capturing all aspects of a distributed, service-oriented application.
Summary
In the first of this two-part blog post, I discussed the general architecture of the Dynasoar framework for the dynamic deployment of services on the Internet. In this second part, I briefly described the prototype implementation of Dynasoar for deploying virtual machines. The implementation used WCF and Virtual Server 2005.
As always, I am interested in your feedback.