AI speeds up software testing. But does it help find the right bugs? Sii Poland publishes the second Testing Lab report 

Sii Poland has published the results of the second edition of its Testing Lab – AI Edition research experiment. The first study showed that AI can significantly increase the productivity of test automation engineers. This time, Sii experts explored a different question: does producing more tests actually mean finding more of the defects that pose a real risk in production? 

AI can help testers build test frameworks, create test cases, and prepare reports faster. But productivity alone does not tell us how well those tests protect a production environment. What matters is whether they can detect the bugs that have real business consequences. 

This question became the starting point for the second edition of Testing Lab – AI Edition. 

This time, AI was not the variable 

The first edition compared teams working with AI against teams using a traditional approach. The results showed that AI enabled teams to produce significantly more working tests in the same amount of time, without reducing average quality. 

The second edition took the experiment a step further. All 16 teams used AI, worked on the same system, and faced the same 19 deliberately seeded defects across different levels of complexity.  

The question was no longer whether AI helps testers, but what determines their effectiveness when everyone has access to AI. 

More tests do not necessarily mean better protection 

The experiment confirmed that AI can significantly accelerate the development of test infrastructure. But the ability to detect defects did not increase in proportion to the number of tests created.  

The biggest differences between teams appeared where technically correct tests were not enough. The tester still had to decide what to test, how to determine the correct outcome, where to look for risk, and when to challenge the model’s suggestions. 

Based on these observations, Sii experts developed the AI Operator Model and 5 quality gates designed to turn AI-assisted testing into a more effective and repeatable process. 

What does the tester’s role look like in the age of AI? 

The new report presents the full results of the experiment and examines what differentiated the approaches taken by the 16 teams. It also explores a broader question: what skills do testers need when AI can generate code, run tests, and suggest fixes on its own? 

The findings point to a shift in the tester’s role. AI can act as an executor and exploration partner, but the tester remains responsible for the quality process – defining what matters, challenging results, and making sure the tests are capable of detecting meaningful failures.  

Read the full report, “AI Testing Lab. Part II: AI Is Not Enough. The Role of the AI Operator in Test Quality,” to explore the data, findings, and practical recommendations from the experiment. 

Download the report and find out what makes AI-assisted testing effective.