FutureGrid and ConPaaS

I am visiting Vrije Universiteit in Amsterdam for a sabbatical, and a major collaboration has been with the ConPaaS team led by Thilo Kielmann and Guillaume Pierre (who's now at Rennes). One of the main activities in our collaboration has been the integration of technologies that are part of the FutureGrid software stack - the IPOP virtual network and the Grid appliance "bag-of-tasks" middleware - with the ConPaaS Platform-as-a-Service system. This has been a very interesting activity, as I get  a chance to learn more about various PaaS technologies, and witness the possible benefits that IPOP/Grid appliance technologies bring to other projects.

First, a bit of background. IPOP is an "IP-over-P2P" overlay that self-organizes virtual private networks across the Internet; the Grid appliance is a virtual machine appliance that encapsulates IPOP and high-throughput computing middleware (HTCondor) to enable easy-to-deploy self-configuring virtual clusters for "bag-of-tasks" applications. FutureGrid users have access to run Grid appliances across its infrastructure for research and educational projects (check out our tutorials for details on how to run the Grid appliance on FutureGrid).  ConPaaS is part of the EU Contrail project; it provides a runtime environment to easily run applications on the cloud by deploying full-fledged platforms as a service. Its applications include Web hosting, task farming, and Map/Reduce distributed computing.

What IPOP can bring to ConPaaS is the ability to create virtual private clouds that span across multiple providers, as well as handling network address translation (NAT) and firewall devices. Essentially, IPOP-enabled ConPaaS services can communicate over a VPN that is decoupled from the public Internet. The concepts behind the Grid appliance are applicable to one of the services that ConPaaS offers - task farming. By encapsulating the IPOP virtual networks in virtual machine images, a task-farming service can transparently aggregate resources across multiple cloud providers into a single virtual resource pool that is scheduled by HTC middleware - for independent tasks, as well as for workflows.

So, there is a nice synergy between these systems, but getting complex software systems to work together is where the rubber meets the road. As a first concrete step in the integration, we had a productive hands-on meeting this month - where we installed and tested IPOP in the ConPaaS image, shook off a couple of bugs/configuration issues, and worked out a path towards integrating IPOP as a virtual network service that is optionally enabled by a ConPaaS user. The initial tests went well, and we were able to connect ConPaaS VMs running across Amazon EC2 and the DAS cluster at Vrije.

We hope to have integration ready by the next release of ConPaaS. In the meantime, the ConPaaS team shows that we "eat our own dog food" to celebrate the release of version 1.1 - now the project's web site www.conpaas.eu can rely on its own ConPaaS technologies to host it. Cheers!