An open approach to code reviews

One of the key aspects of every software product is the quality of the product itself. This is often measured by taking the number of bugs, performance metrics, the amount of memory needed to run it, and shown to the world in the form of attractive reports and graphics. However, there is one aspect that is very difficult to measure but extremely important: code quality. How can this be measured? Measuring code quality can be a very subjective process, so instead of relying on reports and tools to generate cool graphics, the smart approach is to integrate the code quality assurance process into the development cycle. Today I’ll explain how adding a code review process to our development cycle has helped us deliver better software.

It’s all about culture

There are many products that are great at controlling the code review workflow, such as Gerrit or Review Board. They provide a convenient way to review the code, integrate perfectly with CI systems, and enable you to control what can be merged into the source code tree and what should be rejected. This kind of tool works well for big, distributed, or heterogeneous teams, but may introduce too much overhead for small ones. When getting things done is a must, all the bureaucracy generated by these tools may not be worth the benefits they bring.

This was the case at Abiquo. We don’t have a big engineering team, and we all knew we needed some way to improve the quality of the code we committed. We all understood the need for code reviews and the benefits of having other people validate the code before it is accepted, so we opted for a more relaxed approach: using the Github pull request system to commit every single change.

Our workflow

Every pull request has to be validated by at least two members of the engineering team and cannot be merged until it has their okay. Once it has been accepted, the creator of the pull request can merge it.

One could ask what would happen if the creator merged the pull request without waiting for the validation, but the answer is easy: it does not happen. It is all about culture. We all understand the benefits and the need for code reviews and we imposed the process on ourselves, so naturally we respect it.

And who reviews the code? Anyone! All engineers have the power to approve or reject a pull request by expressing their opinion in a comment (but only the creator is allowed to actually merge or close it). There is no formal process that sets the reviewers, and no tool integrated with the CI system that pushes the changes once approved. We just trust each other, and it works.

This kind of altruistic review process works fine if you only have a few repositories to watch, but we have more than 30 different repositories where we commit regularly, so we have built a code review dashboard that helps give us a global vision of what is going on. It shows which pull requests are pending review, which have been approved and can be merged, which are old and should be reviewed ASAP, which have no reviews yet, and so on.

The dashboard is configurable and deployable to Heroku, so you can just download and try it with your Github account.

In just a couple of months since we adopted the code review process, we have seen many benefits:

• There are considerably fewer issues in our internal releases.
• We are all improving our coding skills and learning a lot from other points of view.
• We have a more global vision of the changes in the platform.
• This process indirectly helps the QA team to test without being blocked by minor issues.
• We write better software!

Being a small team, sharing the same culture of software development, and having this open approach to code reviews has allowed us to introduce the process with practically no loss of agility and helped us deliver better software since the first day of its adoption.

Deploying Abiquo 2.4 with Cobbler

Now that you can download Abiquo and get a 30-day trial license, you may like to automate the deployment of a new datacenter in your Abiquo environment. I’ll explain here how to automatically deploy Abiquo using Cobbler.

First of all, you need to get Cobbler up and running. I’m not going to explain here how to install it, but you can use the official guide that is very detailed or these instructions for installing on CentOS 6. However, I will give you some advice about Cobbler installation. Review the output of the ‘cobbler check’ command to double check that everything is working fine. You will need to have at least DHCP and TFTP managed by Cobbler.

(more…)

Locking disks on KVM using libvirt-lock-sanlock

When you are managing a huge number of virtual machines with a shared datastore, it could be very dangerous for different virtual machines to access the same disk file or volume. This can lead to data consistency problems, and in a worse case scenario, the loss of all information on the disk.

We all know that things that should never happen sometimes do. That’s why it’s good practice to add an extra security layer, to ensure that disk access will always be controlled.

A tool for this purpose is listed on the libvirt webpage:

http://libvirt.org/locking.html

With libvirt-lock-sanlock, we create a connection between libvirt and sanlock. Whenever libvirt is using a disk or volume, sanlock will create a lock, so other virtual machines that would be using the same disk won’t be able to start up.

(more…)

Abiquo 2.4.0 EE is here!

It is a pleasure to announce in our technical blog the release of Abiquo 2.4 (internally called King Piccolo). I don’t think it’s necessary to go over all the features included in this release because you can read about them on our public site and I’m sure that Abiquo’s business associates will make an announcement of them on their websites.

However, I do want to highlight some exciting technical solutions included in this release that will move the platform to a new stage of scalability, reliability and robustness (even more so than previous versions).

I think the major change in this release, ahead of other internal improvements, is the full Load Balancing capability of the Abiquo Management Server, enabling you to put Abiquo behind any load balancer and horizontally scale based on the needs of your business.

Implementing load balancing may seem easy, but it isn’t when you have a distributed environment orchestrated by queues, which sometimes needs a coherent job order, or when it isn’t evident when your scheduler engine must maintain the integrity of resource data, so you must ensure one allocation at a time.

(more…)

Addicted to automating complex environments

Welcome to Abiquo, goodbye to the same ol’ thing

We have an important job opportunity available for an excellent engineer who can help us in the continuous integration process, automating the testing and verification of complex situations. Only suitable for bold people.

We’re on the lookout for an Engineer to help us in our quest to build the best cloud management platform on the market. Be part of the Barcelona team and be passionate about building complex Automated tests from building the environments, to driving the software and verifying the results.

 

• Can automate tests that drive not only the Abiquo Platform but the multiple 3rd party platforms with which we integrate (Hypervisors, Virtual Machines, etc.)
• You must enjoy programming and solving complex test automation problems
• Excellent Continuous Integration skills (Jenkins)
• English is a requirement

Desired Requirements

• Experience automating complex environments!
• Knowledge of testing tools,(e.g. Selenium, FlexMonkey..)
• Knowledge of languages that aid automation. (e.g JAVA/C#/Ruby/Python/Perl..)
• Knowledge/experience of Performance/Stress/load/Functional testing techniques
• Source code control experience (Git)
• Experience in an Agile environment. (Understanding of working in a Scrum team – sprints, etc.)
• Good level of Linux administration (Centos/Ubuntu).
• Experience using Eclipse, Maven, JIRA and other Open Source tools
• Nice to have: Certification of a Testing Qualification (e.g. TMap/ISEB/ISTQB..)
• Nice to have: Experience  in Open Source environments (e.g. involved in an Open Source project or community)
• Nice to have: knowledge of virtualization technologies (Xen, VMware, KVM, VirtualBox, etc).
• Nice to have: Chef/puppet capabilities
• Nice to have: Networking knowledge
• An understanding of Cloud Services (Products, companies, etc.) and standards is a plus
• Experience of working in a company producing Enterprise Software

In return you can expect a salary package in line with your experience (Between 34K and 37K + variable) as well as the opportunity to participate in a very ambitious project full of challenges.

Please apply sending your CV to hr@abiquo.com (GitHub account is a plus)

Seeking a Ninja, Rock Star or Borat software engineer

Welcome to Abiquo, goodbye to the same ol’ thing

Abiquo is a Software company with development branches located in Barcelona and the UK, targeting the Cloud Computing market. Currently we are developing a leading IaaS product that is revolutionizing Cloud Management with installations around the world.

We have an important job opportunity available for an excellent Java developer to join the team.

We’re on the lookout for an Engineer to help us build the best cloud management platform on the market. Be part of the Barcelona development team that is passionate about building terrific tools, solving our customers’ challenges and contributing back to the community through different open source projects developed by Abiquo or others, in which we are collaborating.

What we look for in every employee

Get it done
We value aptitude and attitude as well as experience. We value the ability to get things done with the highest standards.

There’s no I in culture
Our company is based around small, dynamic teams working with Agile methodologies. You should work well with others, and treat teamwork and team achievement as the end goal.

You are our ambassador
On a personal level you should have good written and oral communication skills. Every employee is a spokesperson for the company at all times.

Key Experience

Excellent OO development skills, Core Java, JPA, Unit Testing.

Key Requirements

• Can write robust bug-free code that can be reliably used by hundreds or thousands of programmers (i.e., public library-level code)
• You must enjoy programming and solving complex problems
• Excellent Core Java skills
• Excellent OO skills
• English is a requirement

Desired Requirements

• Experience/knowledge building distributed/fault tolerant/highly concurrent architectures
• Experience with REST APIs / JAX-RS
• Knowledge of message systems (RabbitMQ)
• Knowledge of Hibernate/JPA
• Experience with Test-driven Development and automating anything!
• Source code control experience (Git)
• Experience in an Agile environment. (Understanding of working in a Scrum team – sprints, etc.)
• Experience using Eclipse, Maven, JIRA and other Open Source tools
• Experience in Open Source environments (e.g. involved in an Open Source project or community)
• Nice to have: knowledge of virtualization technologies (Xen, VMware, KVM, VirtualBox, etc.)
• Nice to have: Networking knowledge
• Nice to have: WMI, DCOM knowledge
• A good knowledge of cloud market (Products, companies, etc.) and standards is a plus

In return you can expect a salary package in line with your experience (between 34K and 37K + variable) as well as the opportunity to participate in a very ambitious project full of challenges.

Please apply sending your CV to hr@abiquo.com (Github account is a plus)

Developing tools with Abiquo API: Event notifier

One of the KEY points of Abiquo is that you can use its API to access ALL resources and actions available from the graphical user interface. This enables us to create an infinite number of external applications that you can integrate with your environment or platform. All the API resources are documented on our wiki.

In this article, we are going to present the Abiquo Events Notifier, a Python tool that monitors events performed in Abiquo and based on a configurable list of rules, sends notifications by email to the cloud administrators, enterprise administrators, etc. This tool can be very useful for the cloud administrator in tracking errors on the platform. Also, you can configure the tool to notify your users when a deploy is finished or if a virtual appliance they own has been undeployed. You can find the full documentation on Abiquo DevOps wiki

(more…)

Using Abiquo with libcloud

Recently a new libcloud version was released. This is the first time it includes the Abiquo Compute Driver.

Libcloud aims to be an integration library for different cloud providers. It is defined by its easy integration with plenty of drivers and its easy-to-use philosophy.

Libcloud hosts Compute Drivers, Storage Drivers, Load Balancer Drivers and DNS Drivers. We will focus only on Compute Drivers.

The Cloud has become a heterogeneous world and the target drivers range from VPS providers, through public IaaS providers, to private IaaS providers, and there is even an experimental libvirt system library wrapper to manage your local deployments.

This means the libcloud API is generic. So it is important for to you know the limitations and restrictions (and advantages, of course!) you will have as an Abiquo user if you use Abiquo through libcloud, as well as the preparatory steps an Abiquo Cloud Administrator should follow to provide a good user experience. This post enumerates them.

Design Decisions

Conflicts with Abiquo and libcloud

The VirtualDatacenter concept does not exist in libcloud, and neither does the VirtualAppliance. In libcloud everything is a Node. These Nodes are defined by NodeImages and deployed in NodeLocations. That’s all. This is really simple compared to the idea of multiple VirtualDatacenters with different hypervisor technologies that we have in Abiquo. Not all VirtualImages (NodeImages in libcloud) are compatible with all the hypervisor technologies (and hence, VirtualDatacenters) so we have had to take this into account when defining the Abiquo Compute Driver.

More complex: each VirtualDatacenter can store multiple VirtualAppliances and in Abiquo a VirtualMachine cannot be defined outside a VirtualAppliance. This means that handling VirtualAppliances in the Abiquo Driver becomes mandatory, but it has to be done respecting the base libcloud API methods.

And even more: in Abiquo you can configure a VirtualMachine and register it to a Virtual Appliance without deploying it. You cannot do that in libcloud. So Node/VirtualMachine lifecycles are not exactly compatible.

Virtual Datacenters in libcloud

We finally decided (as we did in the jclouds driver) to establish the relationship between VirtualDatacenter and NodeLocation. So when you execute the list_locations() method you are actually retrieving the list of VirtualDatacenters you have access to:

from libcloud.compute.types import Provider 
from libcloud.compute.providers 
import get_driver Driver = get_driver(Provider.ABIQUO) 
conn = Driver('user','password','http://abiquo/endpoint/api') 
conn.list_locations() 
>> [<NodeLocation: id=6, name=Developers, country=Abiquo BCN, driver=Abiquo>,
<NodeLocation: id=9, name=Ignasi, country=Abiquo BCN, driver=Abiquo>, 
<NodeLocation: id=42, name=MyVDC, country=Abiquo BCN, driver=Abiquo>, 
<NodeLocation: id=33, name=Sergi, country=Abiquo BCN, driver=Abiquo>, 
<NodeLocation: id=32, name=Leader, country=Abiquo BCN, driver=Abiquo>]

The context

Because the content of the locations does not change too much, the locations are stored into the session context. When you run the conn.list_locations() you are just printing them.

If you have created a new VirtualDatacenter in Abiquo and want to see it as a location in an existing libcloud session, you should run:

conn.ex_set_context()

to refresh the context.

Virtual Machines in libcloud

Abiquo’s VirtualMachines are mapped into libcloud’s Nodes. We have covered all the data and extra data needed to create Nodes, but if you are familiar with Abiquo maybe you will find the behavior does not match with the expected one because of the name of the functions and Node states. So the standard Node lifecycle in libcloud could be as shown in the following diagram.

In the above diagram, nodes are libcloud NodeStates, blue arrows are user actions, and black arrows are internal business logic to pass from one state to another.

In Abiquo, you can define a VirtualMachine before deploying it, and we wanted to offer a way to use these defined nodes (always defined in another Abiquo client, because the define_node() function is not exposed in this driver… yet). So there is also the function

conn.ex_run_node()

which fits into libcloud’s Node lifecycle as shown in the following diagram.

Virtual Appliances in libcloud

There is no concept of Virtual Appliances in Libcloud, so we created a new object in the Abiquo Driver: the NodeGroup. A NodeGroup is just a logical group of Virtual Machines that can be handled together. For libcloud’s create_node() method, we have added a keyword to specify in which group (Virtual Appliance) we want the node to belong to. So, there is a new method parameter called group_name.

A NodeGroup belongs only to a single NodeLocation, in the same way a VirtualAppliance belongs only to a single VirtualDatacenter

If you don’t use group_name in method parameters:

conn.create_node(image=images[0]) 

the driver will create a node inside the group libcloud (default group name). If this group name does not already exist, then it will create the group. If the group exists, it will add the node.

If you specify the group_name in the arguments:

conn.create_node(image=images[1], group_name='virtualapp')

the driver will create a node inside the group with the specified name.

If you create another node the same way:

conn.create_node(image=images[2], group_name='virtualapp')

but if, due to image restrictions, the libcloud driver decides it must be deployed in another location, the ‘virtualapp’ group will be created again (if it does not exist) in that location.

This means there is no way you can receive some kind of restriction error when you create or define NodeGroups. The aim is to be as flexible as possible but not limit the standard libcloud API. This is the reason why we accept a group name instead of the NodeGroup object itself.

Extra Group Methods

Create a group:

conn.ex_create_group(name='virtualapp')

It creates a new NodeGroup.

Destroy a group:

conn.ex_destroy_group(group)

Destroys a group. Here you must set the NodeGroup object. Take care! It will destroy all the Nodes inside the group as well!

List the groups you have defined:

conn.ex_list_groups()

For more information, please see full driver documentation

Define privileges

The Abiquo libcloud driver is designed to be used by standard users without many privileges in the platform. These privileges are required for full feature access:

• Access virtual datacenters view
• Manage virtual appliances
• Edit virtual appliance details
• Deploy and Undeploy airtual appliances
• Assign network to virtual appliance
• Assign volumes to virtual machine
• Perform virtual machine actions
• Access Appliance Library view

Because an image is worth a thousand words, here is a shot of the ‘Privileges’ screen in our Flex client:

Abiquo API compatibility

The current release is compatible with Abiquo 2.0 and Abiquo 2.2, which are the versions that most of our customers have installed. We don’t think there would be any problems with newer versions, but we have not tested it with Abiquo 2.3 or Abiquo 2.4 yet.

Next Steps

In the current approach, we have tried to be as “libcloud standard” as possible. We are planning to expand the functionality to IP assignment, volume assignment and more Node states (stop, define..), while maintaining libcloud’s standards, in order to improve the user experience and be more helpful to our customers.

More info

• Basic libcloud usage
• Abiquo Driver Documentation
• Libcloud 12.1 changelog

Cooking your own configuration in the cloud

Although configuration is one of the most important aspects of cloud deployment, it is often neglected. There are several automation tools and platforms available, such as Opscode Chef or Puppet, but a minimal infrastructure is usually required to be able to use them. They are really good at automating configuration in large environments, but this is not always the use case in the cloud.

People want to create services in the cloud quickly and easily, without having to worry about infrastructure apart from the services that they are deploying. So how could service configuration be automated? SSH or command-line scripts are often very complex to build and maintain, and making them work on multiple operating systems can be painful. How could this be resolved quickly and easily? The answer is to team up Jclouds and Chef Solo. They’re a perfect match!

Cool features of Chef Solo:

• Run cookbooks on your nodes without a Chef Server.
• Many cookbooks are available, compatible with most common operating systems.
• Configure roles and data bags.
• Schedule Chef runs to keep your node up to date.

Ok, this sounds great, so let’s get into the code!

Cooking a basic web server

This first basic example will install an Apache2 web server on a node. It will download a tarball with all the cookbook definitions and install the Apache2 recipe on the deployed node.

// Create the connection to the cloud provider
ComputeServiceContext computeContext = ContextBuilder.newBuilder("<provider>") 
    .endpoint("<provider endpoint>")
    .credentials("<identity>", "<credential>")
    .modules(ImmutableSet.<Module> of(new SshjSshClientModule()))
    .buildView(ComputeServiceContext.class);
ComputeService compute = computeContext.getComputeService();

// Build the Chef Solo script that will download the cookbook definitions
// and install the selected recipes
Statement bootstrap = ChefSolo.builder().
    .cookbooksArchiveLocation("http://foo/bar/cookbooks.tar.gz")
    .runlist(RunList.builder().recipe("apache2").build())
    .build();

// Select the template to be deployed
Template template = compute.templateBuilder()
    .imageNameMatches("Ubuntu.*")
    .options(runScript(bootstrap))
    .build();

// Deploy the node and bootstrap it using Chef Solo!
compute.createNodesInGroup(group, 1, template)

Have you noticed that there are only 5 lines of code? That’s how easy it is to bootstrap nodes with jclouds and Chef Solo!

Playing with roles and data bags

Chef Solo also lets you define roles and data bags on the fly, to provide more flexibility to your configuration. This example will show how a role can be created and used to customize the Apache2 installation.

Role role = Role.builder()
    .name("webserver")
    .description("Web server with apache and SSL")
    .jsonDefaultAttributes("{\"apache\": {\"listen_ports\":[\"8888\"]}}")
    .runlist(RunList.builder().recipe("apache2").build())
    .build();

Statement bootstrap = ChefSolo.builder().
    .defineRole(role)
    .cookbooksArchiveLocation("http://foo/bar/cookbooks.tar.gz")
    .runlist(RunList.builder().role("webserver").build())
    .build();

That’s it! With just these two lines you can create a role to set up the default configuration for your web server deployment.

Automating maintenance

In the examples I’ve used the cookbooksArchiveLocation to download a cookbook archive. This could also be a local file system path if you are able to upload the cookbooks to the node (for example, using the jclouds SshClient).

Instead of downloading or uploading a cookbook archive to the node, another common use case is to clone a git repository with all the cookbooks and set up a cron job to keep it up to date. Then the Chef Solo statement can be configured to run periodically, and this way the node will always be up to date with the latest version of the cookbooks.

// Clone all community cookbooks in the '/var/chef' directory
Statement cloneCookbooks = CloneGitRepo.builder() //
    .repository("git://github.com/opscode/cookbooks.git") //
    .directory("/var/chef") //
    .build();

// Configure Chef Solo to read the cookbooks from there
Statement chefSolo = ChefSolo.builder().
    .cookbookPath("/var/chef/cookbooks")
    .runlist(RunList.builder().recipe("apache2").build())
    .build();

// Build a boostrap statement that installs Git, clones the cookbooks and
// runs Chef Solo
StatementList bootstrap = new StatementList(
    new InstallGit(), cloneCookbooks, chefSolo);

As you’ve seen here, bringing configuration management to the cloud is easy with jclouds and Chef Solo. There is no need to build complex infrastructures or deployments. Just start cooking your own stuff!

Abiquo 2.2 – Improving ESX capabilities

Abiquo users in Proof of Concept and Production environments send us a large number of requirements, features, improvements and other comments that we evaluate and try to include in our roadmap in order to better cover and fit our customers’ needs.

In Abiquo 2.2, we decided to further investigate and engage with some of these requirements and incorporate them into the product. In particular, the idea of these features is to cover use cases and daily situations that our customers face. In some cases we have taken a first step to helping our customers before incorporating a complete feature in later versions.

These features have been developed for ESX because it is our major installation base. Although, as always, we will incorporate them for other hypervisors in future releases.

The features are: