Software delivery has improved dramatically since DevOps workflows have become the de facto method of deployment all companies aspire to implement. It’s not uncommon to hear of hundreds of deployments a day being pushed to production, with AirBnB proudly declaring they make over 125,000 deployments per year.

However, this level of automation and speed of deployment creates problems of its own—where bugs can easily be missed and pushed to production. This means a well-thought-out testing strategy must be put in place to ensure no bugs slip through the net and degrade the quality of the application.

In this article, we will review the role of testing for DevOps engineers, the main types of tests that need to be carried out, and tools to help you while building strong testing pipelines.

The Role of DevOps in the Testing Cycle

An essential aspect of testing is identifying the roles of a tester and the reasons for testing—unlike other development processes, testing is incorporated at every stage of the CI/CD pipeline.

It falls on the DevOps engineer to ensure that the software is functional and capable of performing its purpose at the time of delivery. This does not mean that a DevOps engineer must write test cases for each deployment (which usually falls on a developer or tester), but a DevOps engineer must make sure that the tests are executed efficiently and safely, so any issues can be highlighted and prevented from reaching production.

Types of Tests to Execute

There are many types of tests that may become part of the CI/CD pipeline, but as a rule of thumb, if it can be automated, then it should. First, we’ll focus on some of the core types of tests that are commonly executed in a CI/CD workflow.

• Unit tests are automated tests that are executed on the source code level—they test small units of the application.
• Functional/integration tests are manual or automated tests where a user interacts with an application to make sure it does what is expected from it according to its specifications.
• Acceptance tests are automated tests that are executed on the application level. They test if an application meets its specifications, but focus more on how it is intended to be used rather than what units of code are involved in producing a certain behavior or functionality.

Some other forms of software testing that may be incorporated into a CI/CD workflow are load testing, performance tests and security audits, but for the purpose of this article, we will stick to the three mentioned above.

Unit Testing

These tests aim to verify if individual parts of the code are working as expected. Unit tests can be written by developers, testers or business analysts, but they must be executed automatically within an automated testing framework like JUnit (for Java), RSpec (for Ruby) or PyTest (for Python). A good practice also has a unit test coverage report of at least 80 percent, to ensure the core functionality of the application is working as expected.

Executing unit tests is one of the easier parts of the CI/CD workflow to operate as all it requires is the necessary runtime of the language to be available, and an ability to recognize when a test fails.

As many teams use GitHub as a central repository for their code, GitHub Actions is a good place to implement unit tests. For example, you can set up a workflow to run whenever a branch is merged into the test branch before it is pushed to the test environment. This allows a developer to run a test on their code very quickly, and see if it will pass or fail, thus preventing it from reaching the next stage in the deployment cycle.

Functional/Integration Testing

The aim of functional, or integration testing is to ensure that the various components of your application work together correctly. Where unit tests take place in relative isolation, the aim here is to test that different systems can work together to provide the core functionality that is expected.

This is especially important in a microservices world where integration between systems is more important than ever. Some common tools to automate integration tests in a CI/CD workflow would be Cucumber or Selenium Webdriver, with DX Infrastructure Manager in place to monitor any irregularities between services.

As running integration tests often require multiple systems to be running, it is difficult to carry out these tests as part of an internal workflow as we saw with unit tests. Instead, these tests might take place in a staging environment, where the tests are run at the end of the deployment process, making sure any issues are detected and if not, then pushed to production.

Acceptance Testing

In addition to functional testing, a parallel workflow should be run to test if the software actually meets the required use case. Where integration tests are designed to check that the system is functioning as expected, it is the role of the acceptance tests to ensure that the output is correct. In some cases, this means that the system is performing exactly as expected.

In other cases, it could mean that a certain amount of error or variety may be present in the output for test purposes. For example, an API should return an error if the input data is invalid.

Acceptance tests are often used to describe how an end-user will interact with something and what they should expect from each action taken within a software application, that is why it is core to stay as close to production as possible, so running these tests as part of the CI/CD workflow on the staging environment is a good idea.

Creating and maintaining a strong acceptance test workflow can be extremely time-consuming and quite daunting, so tools like Test Data Manager by Broadcom Software can automate and help out with test data management and storage. By staying on top of your test data, you can ensure that tests are repeatable and explainable so any issues can be found and remediated.

Some Additional Things to Note

Make Sure Everyone is on the Same Page

As your teams grow, and each may be working with different technologies, make sure each team uses a known testing framework so it can be used with existing workflows. As your workflows become more automated and numerous, you don’t want to spend any time on writing new workflows because a new team is using a different framework.

Use Test Automation Frameworks

Test automation can help ensure that your tests are executed in an automated fashion, and will create more reliable results over time. Automating testing ensures that at least some level of regression testing is done every time a change to the system is under development. Sometimes it is tempting to write your own testing software, but don’t reinvent the wheel! Test-driven development has been such a popular practice for a number of years now that there are plenty of frameworks out there to help you.

Closing

Creating a strong testing workflow can seem like a herculean task at first, but if executed correctly with the right tools, the overall quality of your software will steadily improve, along with the frequency of deployments. Contact A&I Solutions to discover how our DevOps solutions like Test Data Manager by Broadcom Software can help speed up and simplify your testing process.