I recently explained how testing everything in OpenStack is hard.

The reality is that the RDO community has finite resources. Testing everything is not just hard, it’s time consuming and expensive.

I’ve been thinking a lot about how to tackle this and experimented around ways to improve our test coverage in this context.

How can we improve testing coverage for RDO ?

Upstream OpenStack does a good job of testing things as much as it can.

Projects like Puppet-Openstack and Kolla will gate every commit with integration tests to make sure that nothing is broken within their project and these tests happen to consume packages provided by RDO.

As a matter of fact, they’ve managed to find some bugs that the RDO CI had missed because they test other projects, configurations or implementations that we don’t currently have coverage for. We embarassingly apologized (sorry again guys!), thanked them and worked on resolving the issues.

So that got me thinking… We have these upstream projects doing integration tests inside the OpenStack gate and testing infrastructure. They each have their own opinions about how they install and configure things and they make sure that it works when consuming packages provided by RDO.

It’s great because these are entirely different ways of installing and deploying OpenStack.

What if we could take those tests and run them outside of the gate on packages even before they land in stable repositories ?

Introducing WeIRDO

WeIRDO stands for “We [ Install | Integrate | Improve ] RDO".

WeIRDO is an idea that is about leveraging the effort the community puts towards their own CI initiative to improve the coverage that upstream RDO provides. This is about testing as many ways of installing, configuring and integrating OpenStack as possible without re-inventing the wheel,

WeIRDO takes the gate jobs outside of the gate to test what RDO ships before it’s packages even make it to the upstream OpenStack CI.

When projects that deploy OpenStack with RDO packages improve their integration tests, WeIRDO will automatically pick them up and run them before pushing packages to the stable repositories.

This can also mean that WeIRDO jobs can unintentionally break but I can see how we could have third party non-voting gate jobs on implemented projects to give feedback about their patches.

Everyone wins:

• RDO has increased test coverage and submits bugs upstream when necessary
• Deployment and installation projects are more stable and productive
• Vendors, operators and end users benefit from a better experience

How does WeIRDO work ?

In a nutshell, WeIRDO is a collection of Ansible roles and playbooks.

WeIRDO, through Ansible, will:

• Create an ephemeral node on which the CI job will be run
• Generate a similar environment to the gate (setup configuration, packages, dependencies)
• Run the gate job (i.e, run_tests.sh or tox provided by the upstream project)
• Destroy the ephemeral environment

This is not unlike what the Openstack Infra team is already doing with Jenkins and NodePool - and with good reason. We don’t want to be too different than the gate to involve as little implementation effort as possible.

WeIRDO, by itself, doesn’t use or require Jenkins. When run inside a job, however, it will be able to report success or failure.

If you’re interested in the technical details, feel free to browse the documentation at http://weirdo.readthedocs.org/en/latest/.

How can you help ?

WeIRDO is a brand new open source initiative and is available on Github at https://github.com/redhat-openstack/weirdo.

Contributing is easy.

It’s meant to be by the community for the community.

Here’s some examples of what we could use some help with:

• Integrating other targets for test environment deployment (supports only ci.centos.org right now)
• Integrating other installers that consume RDO packages (Fuel ? Openstack-Ansible ? Others ?)
• Making sure gate jobs can be run outside of the gate (Some upstream patches were required to make this happen because of some gate assumptions)
• Improving documentation
• Checking in on Jenkins jobs and reporting issues
• Ideas, suggestions, comments, feedback !
• More !?

We hope to hear from you, any feedback and contributions will be highly appreciated !