Manual Testing in the Age of AI: Evolving, Not Obsolete

Did AI Replace Human Testers? Five Years Later, Here’s What We Know

Almost 5 years ago, I wrote a blog about whether or not Machine Learning and AI would replace humans who did manual testing. Machine Learning and AI growth have exploded since then. A big part of that has been the adoption of AI tools in development and testing. My opinion has not changed. AI and Machine learning won’t replace a Human Tester. (In my original blog, five years ago, I used the term “manual tester". In the context of partnering with AI tools, I use “Human tester” instead.)

In 2020, Machine Learning and AI lurked in the shadows. People knew that they were a thing but most people had limited experience with either of them. Google Maps, Ride-Sharing apps, and chatbots were their major exposure to them. Five years later, experience with AI and Machine Learning is more common. Many people use ChatGPT, Claude, Perplexity, and Microsoft Copilot, to name a few. Rather than IF Manual Testers will use AI, we should ask HOW they will use AI. Part of the answer to that question means learning new skills to work with AI tools. 

Haven’t Agile and automated testing already made manual testing obsolete?

No, not at all. Automated testing and Agile, reshaped the skill sets for Manual Testing. With Agile, we learned to test based on user stories and not bulky requirements documents. We got involved earlier in the process. We learned to collaborate with Developers and Product Owners rather than working in silos on our own.  You can read more about that here.

What do manual testers need to understand about AI?

Testers need to understand why AI matters in QA right now. Test automation driven by AI has increased. AI is also used in predictive analytics, defect detection, test generation, and optimization. There is a competitive advantage for QA professionals who adapt to using AI. In order to gain that competitive advantage, Human testers need to be familiar with the core AI concepts. You will need to know the basic terminology such as Machine Learning, Deep Learning, Neural Networks, and LLM’s. An understanding of Supervised vs Unsupervised Learning is important to understand how AI learns from data. This will be crucial for generating test cases and predicting defects. An awareness of Natural Language Processing (NLP) will aid in understanding how tools like ChatGPT or Copilot assist in writing and interpreting tests.

What are the new technical skill sets necessary for AI?

Human testers need to learn new technical skills before they can work with AI tools or test AI or LLM’s. The good news is the natural curiosity of a tester leads them to adapt quickly. Let’s get into what QA Analysts must know to get value out of AI. The QA/Testing role will need to evolve to work with AI to deliver quality products. Skill sets include:

1. Data Literacy
   • Understand datasets, features, and how data affects AI models
   • Understand cleaning, labeling, and curating QA-specific data for testing
     
2. Prompt Engineering
   • Learn how to write effective prompts for LLMs to generate or improve test cases, automate bug reports or explain complex code
3. Test Strategy for AI Systems
   • Designing tests for non-deterministic behavior
   • Validation of AI model outputs and fairness testing
   • Scripting & Automation
     
     • Python (most AI tools use this language)
     • AI testing libraries  (PyTest + AI plugins or Selenium with AI layers)
   • Model Evaluation Techniques
     • Basics of precision, recall, F1 score, and confusion matrix in the context of AI tool effectiveness

What Soft Skills are Necessary for QA and AI to work together?

AI tools can automate repetitive rule-based tasks. They can reduce the time it takes for test execution and regression testing. AI tools can also recognize patterns in large datasets. AI tools don’t have the context and domain knowledge that a human has. They can’t do ad hoc, exploratory, or usability testing. They don’t have empathy for users. AI cannot say whether or not requirements were truly met. AI can only let you know if a test passes or fails. 

Human Strengths and Weaknesses

Humans are capable of critical thinking. Human testers have an understanding of business logic and insight into the user’s needs. They can create edge cases, do exploratory and ad hoc testing. Most of all, a Human tester can ensure quality before testing starts. Unlike AI tools, which focus on test execution or defect tracking, Human testers are involved in the creation of acceptance criteria. They can interact with Developers and Product Owners to get more information that will help them to a better job of testing. 

What could the distribution of work look like for an AI and Human QA Team?

The Healthy Skepticism of a Human Tester

Humans can be skeptical about one-off results and create strong test plans that will identify anomalies and capture consistent results. Regression toward the mean has an important role in AI and Human test teams. Testing AI models may see unusually high or low accuracy on a test set. Don’t stop after either an unusually bad result or a very good result. A significantly bad result could be because of an edge case. A really good result could indicate that the model got lucky on a familiar pattern.

A Human tester will trust but verify. A new AI testing tool may find a high number of bugs in its first run, but in the next run there are less bugs. This may be an example of regression toward the mean. The first run just hit more defects by chance.  A Human Tester needs to be involved with this testing. The Human will know to use multiple test cycles and analyze the results.

If you use AI to detect performance or security issues, you may see that sometimes the system flags an extreme outlier. After review, you see that the next few data points regress towards the normal. Again, a Human tester needs to analyze the results. We don’t want to build thresholds on only the outlier result. Testing AI outputs such as chatbot responses or quality of image can lead to varying results during the first round of testing. Unusually bad or good results may move to average results in later rounds of testing. 

AI Along with the Tester’s Mindset

The mindset of a tester is a necessity when using AI tools for testing. We talked about healthy skepticism earlier in this blog. Skepticism IS NOT pessimism. A Human tester knows we need to prove a feature works. We have a natural curiosity about how things work so we will ask questions and observe how the system reacts. Human testers also have emotional intelligence that AI will never have. Emotional intelligence is an asset in interactions with other Humans that we will need to work with, like Developers or Product Owners. Human testers are life-long learners. We are open to learning. We have to stay current with new testing methodologies. We must also observe so we can adjust how we are testing, based on our observations. We have the ability to jump right in and try to figure out what is going on with a particular feature or system. If I am being honest, a Human Tester will find ways to try to break the system. AI won’t be able to do that.

In Summary

AI will not replace Human testers. Five years later, we know that AI tools are accelerating and optimizing the testing process. However, we also see there are limitations that would prevent using only AI for testing. AI doesn’t have the critical thinking skills, domain understanding or emotional intelligence that Human testers have. Human testers are going to need some new technical and soft skills to partner with AI. That won’t be a problem since Human testers love to learn and have a natural curiosity. The collaborative approach between AI and Human testers will lead to better testing outcomes. Let’s check back in another five years and see how the relationship between Human Tester and AI has progressed.