Automation Testing in CI/CD Pipelines: Types and Stages

Get a rundown of the stages involved in testing as well as best practices and tools to integrate it within your CI/CD pipeline.

In this article, we will address integrating tests in a continuous integration/continuous delivery platform.

Let’s start with the basics first.

What Is Automation Testing?

Software testing refers to the execution of tests according to a set of test cases and comparing the results of actual and predicted outcomes. Several steps and testing techniques are followed in the process. Testing is essential to ensuring the quality of the product. This process is typically done manually by a team of testers. However, in automation testing, the process is automated using software tools and technology. Here, rather than resorting to a manual effort, scripts are created and test cases are run automatically. Automation testing eradicates many problems such as human error, coverage area, etc. It also saves time and enhances the ease of conducting tests with increased efficiency and effectiveness.

Automation Testing in CI/CD Pipelines

Automation testing is a core part of CI/CD pipelines because tests that run fast can provide early feedback to the developer. A problem or a bug that is identified early has the possibility to be rectified early. Therefore, the released product will be more accurate and bug-free. This enhances the overall quality of the product, which garners customer satisfaction. To put it succinctly, the advantages of automation testing in the CI/CD pipeline are as follows:

  • Aligns with the CI/CD idea of “build fast, fail fast”
  • Reduces manual effort, which saves time and the possibility of error.
  • Test results are more accurate with the increased number of test cases and can and cover a wider area.
  • Get immediate feedback on any problem
  • Multiple test results can be generated and compared to ensure quality and consistency

Types of Automation Testing Used in CI/CD Pipelines

  • Unit testing: It is a low-level test undertaken after a module has been coded and reviewed. Test cases are designed to test individual components. The purpose is to make sure each component works the way it is supposed to under any circumstance.
  • Integration testing: Integration refers to testing the interaction of components within the application. This testing is carried out after all the modules have been unit-tested. The primary objective is to test the module interfaces and check that there are no errors in parameter passing when one module invokes the functionality of another module.
  • System testing: Tests are designed to validate a fully developed system and make sure it adheres to the requirement specification document. Typically, at this stage, software is ready for use by potential users. These kinds of system tests are called alpha (carried out by a team of developers within the organization), beta (performed by a selected group of users/customers), or acceptance testing (performed by users/customers to determine acceptance of delivery of the system).

What Are Continuous Integration and Continuous Delivery?

In simple terms, continuous integration allows development teams to integrate their code into a shared repository. This helps maintain code quality and identify potential issues with the local version of the code at an early stage.

Continuous delivery is often called “Continuous Deployment” as well. Everything that is continuously merged by the development team is continuously deployed to the live environment.

Since most developers work in parallel, continuously integrating their code into one repository would mean that the master branch is continuously updated with new features. To ensure that there is no compromise in the code quality with so many changes happening rapidly, testing must happen at the same pace.

It should be no surprise that manual testing in this environment would not be the best approach to achieve this. Automated testing is the key to successful testing in a CI/CD pipeline.

9 Continuous Delivery Stages

  1. Develop: The developer builds the code according to the project requirements or the feature requests.
  2. Writing tests: Once the code is written, tests need to be written. At this point, these tests are usually unit tests written by the developers.
  3. Local Testing: This is then locally tested to check whether all the tests pass and to ensure the code does not break. Often, a percentage is set as the pass rate that the tests running need to meet.
  4. Rebase and Resolve conflict: In an actual development scenario, there will be multiple people merging their code. Developers need to make sure that their branch is updated at all times. Updating the branch with the latest merged code is called “rebasing.” Once it’s rebased, there will likely be some conflicts that need to be resolved. After that, the tests are run again against the rebased code.
  5. Commit: Once the tests have passed, the code is then ready to be committed with all the changes.
  6. Build: The source code developed is then combined to build a deployment artifact that can be run on an instance, like the server if the environment is on-premises. This code is now ready to be deployed to different testing environments.
  7. UAT: The code is then deployed to a test server where testers start to test the feature. These tests can be automated as well as manual.
  8. Merge: If the commit that’s under testing is approved by the testers, this is then merged into the master branch.
  9. Production deployment: Once the code is merged, it is then deployed to production.

The above process needs to be done with every build coded by the developers.

Where Does Automation Testing Fall in This CI/CD Pipeline?

Automated testing ideally happens once the build stage has been completed and the code can be deployed. Unit tests, UI tests, and integration tests can all be run at this stage. These tests help in ensuring that the code meets a standard of quality.

This phase can last from a few minutes to a couple of hours depending on how the automation is architected.

Tests can be run in parallel to execute them more quickly. If a code fails during the test phase, the build can be rejected without further investing in any manual testing time.

Tools Used for CI/CD

  • Jenkins: Jenkins is an open-source tool that is used for continuous integration. It’s free to use and jobs can be configured both by the interface as well as scripts.
  • Travis CI: This tool is free of cost for open-source projects, hosted by GitHub.
  • Gitlab: Gitlab is a version control tool that has its own cloud-based CI methodology. It is supported on multiple platforms which have both free and paid versions.
  • Bamboo: Bamboo is a CI tool by Jira. If your organization uses Jira, then it would be beneficial to check this tool out. It supports automated merging on approval of tickets as well.

Best Practices for CI/CD Pipeline to Make the Best Out of Test Automation

  • Incremental changes: It is always advisable to follow a feature-by-feature approach. If the feature is really big, it is good to break it down into smaller and quicker-to-test features. This is important in terms of automation because if there is an issue, it is easier to figure out the root cause. If your commit is too big, isolating the cause of an issue would be a tough task.
  • Identify what can be automated: It is very common for teams to dive fast and say, “Let’s automate everything,” but this is a common mistake. We must know the purpose of automation and identify the test cases that should be automated.
  • Parallel Tests: Tests should be run in parallel to make testing more efficient and timely. It can greatly reduce the time taken to run tests and thus give the results much faster. But it’s not sufficient to just execute these tests in parallel; it is also important to scale the server size where the tests are running in order to actually make them faster.

Conclusion

Automating tests is an important part of the successful deployment of projects while maintaining a standard of quality. Ensuring tests are run at every stage gives good transparency on the quality of the code. Bugs can be discovered at an early stage, and any delays that might be caused by them can be addressed promptly. Having a CI/CD pipeline with integrated tests helps in speeding up the testing and deployment process.

This article was written by: Joyce Prabhakaran

Quality Assurance Approach and Best Practices

Practicing Agile methodologies means approaching testing in a proactive manner the includes developing tests at the inception of the project.

You know it’s vital, I know it’s vital, so let’s move on and dive right into our recipe for success. True quality comes from quality-minded people, and as such, we’ve subscribed to a different angle towards QA, one that differs from traditional practices.

Approach

Traditional Quality Assurance often involves extensive work and coding before any testing is actually done. This results in finding more bugs in the software closer to the delivery date. At mobileLIVE, our QA involves Test Automation coupled with Continuous Integration from the start. This approach uncovers the majority of bugs at the beginning of the software development cycle and fixes them as the cycle progresses. The result is fewer bugs to reconcile at the end of the project, which in turn, allows for seamless and easy delivery.

Whether you are working in an agile environment or taking a waterfall approach, the overall strategy for high-quality QA remains the same:

Best Practices

Our approach to QA is what truly sets us apart in an already dense development landscape, and we would like to share with you a comprehensive list of best practices that have proven effective time and time again:

Focus on the User Experience

Testing at all levels of the project is crucial, but the most important thing to consider is the user experience. If the UX is bad, it inevitably means the quality is not at par. Although this will likely result in changes during earlier levels of development, the user experience must always be at the forefront.

Automation and Continuous Integration

Automation is vital because it provides fast feedback, enabling continuous integration – a crucial component of the agile methodology. Automation is also key to reducing costs and improving the efficiency of your team, which is one of the main reasons why QA managers must educate themselves and be willing to implement automation practices.

Test Coverage and Code Coverage

Many QA engineers talk a lot about “test coverage,” which gives a good general picture of the quality of the application. However, to achieve true quality, both test cases and code coverage analysis must be considered. For example, even if you achieve 100% test coverage, you need to still aim for at least 90% function code coverage to ensure the best results.

The Shift-Left Approach

Typically, testing starts once the coding phase is complete. However, for optimal quality in an agile methodology, you need to shift the focus and begin monitoring and testing right from the inception of the project. This ensures that bugs are detected sooner, which not only saves time and money but also helps ensure a good relationship with developers on the team, accelerates the delivery of the product, and allows for greater test coverage.

Smart Testing

Complete and comprehensive testing can sometimes present a challenge since many teams may not have the time or resources to go through every possible combination of cases. However, you should be smart in how and what you test. This means, communicating with the developer to find small test cases that will uncover the greatest number of bugs.

Bug Prevention

QA engineers are trained to catch bugs; however, a resourceful QA engineer will also think about how to prevent them in the first place. Traditional QA starts testing at the UI level, but an agile method starts with unit and functional testing and then moves to the UI level. This approach prevents bugs from entering higher levels of development which can cause bigger problems later on and likely delay delivery.

Quality Over Quantity

Focus on major critical bugs and glitches initially rather than several smaller ones.

Agile Testing Process and Approach

Agile is undoubtedly the preferred software development methodology for today’s developers. Agile testing uses the same principles to encompass all elements of software testing from unit to system testing. We recommend using The Four Quadrants of Agile Testing in order to not only meet but exceed quality standards in all aspects of a project.

The Importance of Test Automation

At mobileLIVE, we practice a culture of automation and thus believe that every development team should be automating all tests when possible. There is no excuse for not automating these tests, as in most cases, doing them manually will cause considerable trouble, lead to additional costs, and create untimely delays.

Regardless of how thorough your manual testing is, human error will always be a factor, which is why automating all possible tests is the best way to ensure your results meet and exceed the quality standards you were hoping to achieve.

The Value of Continuous Integration

In addition to automating all possible tests, Quality Assurance managers must also ensure that all feedback is subject to continuous integration. One of the major benefits of automated QA is the fact that feedback is available immediately, which is why implementing that feedback and fixing bugs at all stages of the development cycle is imperative. Build Verification Tests (BVT), and Smoke Tests should be built into every stage of the project as they are the most comprehensive way to detect problems early on and prevent issues in future builds.


Key Takeaways

Most would admit that agile development is the future; however, in order to exceed client expectations and improve the user experience, regardless of what method you choose, extensive QA isn’t a “nice to have,” it’s imperative. As the development landscape grows denser, our approach with its focus on automation, continuous integration, and the other best practices mentioned earlier will help you to maintain and surpass quality, ensure the highest satisfaction upon delivery, and provide the greatest experience for the user, as well as your development team, as possible.

How do you approach QA? We would love to hear it!

This article was written by: Joyce Prabhakaran