With the widespread adoption of DevOps and the goal of fast, frequent deployments to end users, a new approach needs to be taken when it comes to avoid delaying features reaching end users and hindering the DevOps process.

Manual testing is at odds with DevOps

DevOps is broad in scope, but some key elements include automation and Continuous Integration and Delivery (CI/CD). These are not conducive to traditional Quality Assurance (QA) methods such as manual testing or handing code over to a QA team to take live as part of a larger release. The act of handover to a QA team slows down the path to go live and delays feedback to developers who need to wait to hear from the QA team before continuing. All of this is at odds with the goal of DevOps: fast, frequent deployment to end users. It also requires the QA team to fully understand the deployment approach. Often deploying features as a package increases the risk of a break in service when pushing through to production. The CI/CD process focus on a simple pipeline of commit, build, automated testing and deployment with a rollback in the event of a failure, which places the end user at the centre.

Achieving frequent CI/CD

Trunk based development is popular with high performance teams. This forces developers to create a solid set of automated tests to reduce the risk of frequent deployments and changes to the master codebase. High performing teams will release several times per day which requires these changes to be fully tested. The only way this is manageable and scalable is with a full set of automated tests and rollback should something fail. This addresses the challenges of running highly available applications. While traditional QA teams would not be able to keep up with this pace resulting in changes backing up and slowing down the performance of the team.

Shift left on testing

In DevOps, we talk about a shift-left on security where developers take more responsibility to deliver code that meets security best practices; removing the traditional approach of adding a layer of security at the edge. We should apply the same methodology to testing. Developers should take the lead on creating the tests, with QA teams providing support and guidance on best practice.

Developers need immediate feedback when running tests locally by having small unit and component tests for each piece of code. The same tests can then be used in the CI/CD pipeline. Developers should not rely on unit tests at the end of the development cycle.  Instead, developers should look to harness the power of more featured test suites to automate. For example, Functional, Regression, GUI, Performance and Security testing can now be run, fully automated in the build process without adding long delays as have been traditionally seen.

Fail-fast

The application needs to be closely monitored once deployed with any issues triggering an automated rollback to the previous version. Each test pack in the build process that fails needs to stop deployment and notify the developer, providing detailed information on the failed tests, the associated stack traces and logging information. This accelerates the development cycle by providing immediate feedback rather than waiting for the QA team to provide information. This also allows the developers to start fixing the issues that arose immediately  rather than needing to trace through log files or replay the steps the test team took again.

How does this affect quality assurance?

QA still play an important and valid role, particularly in ensuring the overall quality of the application improves. By automating most of the routine testing, QA are able to provide more value by developing detailed technical domain knowledge about the system, focusing on edge cases and areas which are difficult to automate rather than spending time on testing small changes or build errors. They also play an important role in advising the developers on best practice and being involved with the technical impact of planned changes or features. QA shouldn’t be treated as a gateway or a silo but still fully integrated into the development team.

It’s certainly true that DevOps is eating the manual testing and traditional approach to QA. Traditional testing methods simply cannot keep up with the frequent deployments and ever increasing reliability and security requirements. However, there’s still an important place for QA within the development team and it’s now a far more important one to ensure automation testing is done correctly, is following best practice and in focussing on more difficult test cases which help to drive more reliability and add value to the application.

6point6 has a wealth of knowledge and experience in helping organisations to develop, embed and scale their DevOps capabilities.

Get in touch if you’d like to chat to us.

Written by Jamie Whittingham, DevOps Consultant