The Ultimate Guide to Pipeline Smoke Testing
Smoke testing (sanity or build verification testing) is critical to any CI/CD pipeline. After major or minor code changes, it helps ensure the system works as expected.
Adding smoke tests to your CI/CD workflow will save you time and effort by automatically discovering new bugs before they’re exposed through regression or acceptance testing.
What is a Smoke Test?
Smoke tests are broad, shallow functional tests for a product that are run early in the testing cycle. They ensure the core software features work and can be used to discard a bad build before more rigorous tests are performed.
The key is to create a comprehensive and well-defined set of smoke test scenarios covering your application’s most critical functions. These can be manual or automated, and they should tell you whether your software is ready for more rigorous functional testing.
Getting Started with Smoke Tests
Smoke tests are a minimal set of test cases that QA teams perform on each build to verify basic functionality. This allows them to discover simple failures before proceeding with other, more thorough testing. Without smoke testing, significant issues could slip through the cracks and impact a product’s overall performance.
In data pipelines, smoke tests ensure that each new build contains no functional regressions. They can also be used to confirm that the data pipeline is appropriately configured and producing the correct results.
To start, identify your application’s most critical user flows and write smoke tests for them. These should combine functional end-to-end tests, unit tests, integration tests, and API tests. Once you have a test suite for your smoke tests, automate them and perform them regularly throughout your software development, testing, and release cycles. This will help you reduce the risk of defects making it to production and limit the amount of time needed to catch them in the future.
Creating a Smoke Test Suite
In addition to providing a simple way to validate basic functionalities, smoke tests can also be used to identify serious system problems that can’t be detected by traditional sanity testing. This can help teams avoid wasting time by identifying the need for more intensive testing before distributing a build to the broader QA team.
A typical smoke test suite can cover a range of abstraction levels, from the overall functionality of a software product to specific sub-systems. It can even include tests for individual data assets in the pipeline, such as the ‘raw_country_populations’ data asset. In this case, a smoke test would verify that the source data is of the right shape to be consumed by downstream data transformations.
Ideally, the test will automatically construct empty data frames using the column schema metadata for each data asset in the pipeline. This will enable you to run the tests without overwriting your production database and causing unnecessary downtime.
Performing Smoke Tests
Smoke tests can be run during local development to speed up iteration and in CI to prevent bugs from making it into production. You can smoke test individual transformations, all transformations downstream of a particular change, or your entire data pipeline.
To perform smoke testing, QA testers must decide which tests they want to include in their test suites. They should then create those tests (manual or automated) and record the pass/fail metrics. As a best practice, it’s recommended that a single script is used for each smoke test to increase flexibility.
Smoke tests are a quick way to ensure a new build has no show-stopping bugs. However, they should not be used as a substitute for more thorough functional and integration testing. Ideally, they will be used with these more extensive tests to help the software testing process become more efficient. In addition, they will help identify when additional sanity testing is needed.
