At Infraheads, we have worked with the QA engineers the most. Failing CI pipelines we implement mainly fail because of the flaky tests. In most cases, the CI pipeline users are not prepared to reflect on error messages shown in the pipeline logs, and we get the blame. That leads to many problems, especially in the teams that try to practice SCRUM. SCRUM masters assume they have a DevOps resource, but that resource is forced to do a reactive job most of the time instead of implementing robust tools for improved SDLC and IaC. That is why Infraheads DevOps agency separates support from implementation.
Based on our observation, most software engineers underestimate the importance of quality tests, and most QA engineers need to be more skilled to implement a quality testing codebase. The lion’s share of flaky tests is due to implementation, not the infrastructure. But there are other problems as well. Ignoring proper reflection on the “Testing pyramid” is an even bigger problem.
One of our clients was in the middle of splitting a 10-year-old C# monolith, but they still had over 10,000 tests. The execution time was over 3 hours, and the monthly Azure bill was over $1 million. Before production deployments, the pipeline had been executed multiple times until each test was passed at least once. They had a dedicated team for that, and it turned out that they were not unique at all. This situation results from a need for a common language and understanding of test types. It is easy to find a folder in the Git repo called unit tests with integration tests or vice versa. In many cases, the same mistake of letting the developer implement enterprise application tests, pipelines, and everything else shoots back.
There are multiple versions of the testing pyramid, but we prefer the simplest one shown in the image.
TDD is an excellent idea. However, some 10x developers would argue about how they want to use the tests and are right when they question any widely spread practices. For most companies, unit tests should be implemented before the feature implementation based on the document or formal requirements represented as tickets or something else. We are shy about saying documentation as Scrum questions the importance of documentation, and we need more confidence to argue. What we know for sure is that quality communication is essential.
The rule of thumb is that having as many unit tests as possible is a good sign, as unit tests are:
Executed within seconds
Cost-effective
Keeping the infrastructure simple
Less flaky
These are just a few benefits of unit tests.
Integration tests are expensive and make the codebase complicated. Mocks are used for cost-saving and efficiency, but they create silos and contribute to pushing the bugs down the line. One exciting idea is Pactflow, which is an implementation of contract testing. Contracts are outstanding; one of the fundamental decisions made by AWS is having clear regulations between the services; contracts help to achieve that, but we remember that it is not about the tools; it is all about the people. Pactflow helps achieve integration testing without spinning up an entire infrastructure. It makes integration testing an asynchronous mechanism instead of the classical synchronous approach. This makes the integration testing much faster but less reliable, and it takes skill to benefit from it.
End-to-end tests should be executed only before the release on the staging environments. If an organization runs those tests significantly more than the deployment pipeline, its processes probably need to be corrected.
Comments