Test Execution: It's Techniques With Best Practices

OVERVIEW

Test execution techniques include planning and strategies for improving test execution and, in turn, the overall testing process. While it affects the test execution, it has no impact on the actual "running of the tests.”

In recent years, test execution techniques have been overshadowed by things initially considered more valuable, such as applied methods and tools used in the process. The reason for this is how testing has evolved over the years. We started with manual testing, but when things started to become repetitive, we turned towards automation testing and worked ahead writing scripts.

For this, we needed tools that then became so popular that different organizations and communities of developers started bringing new and unique features to the table. Some could replay the tests, while some could run tests without any SDKs..

The tools come into the picture in the part where we execute the tests. They do not tell us which ones to perform at what time or what type of grading a tester needs to do. They do not even tell us what test strategies to follow for test execution to gain maximum benefits. These methods and tools give us the framework, but the direction is purely done at the tester’s will.

Test execution refers to the execution of the test code and comparing the actual and expected results for analysis and reporting. For people not associated with testing specifically, test execution refers to the real meaning of “testing” as it includes something that looks “meaningful” because we see test cases running on the application.

In reality, test execution results from all the previous combined stages. These stages include test planning, reviewing, and analyzing by QAs and test design manually or through test design tools. Once the code execution is completed, we move on to other phases, such as reporting, mapping, and re-testing. All this combined is referred to as test execution in the testing field.

Test execution is the execution of test code on the application under test and done using test execution tools. So, if the tool works as an efficient machine, what do we mean when talking about test execution techniques?

A test execution technique does not relate to the test execution part at all. It does affect and gets affected by test execution, but the actual “running the tests” part has no role to play. A test execution technique is our plans and strategies to improve the test execution part and, in turn, the overall testing phase.

In this section, we explore different types of test execution techniques.

When we talk about results (or other such things) in binary, we often ignore a few areas that are neither entirely on this side nor there. The best example to quote here is “Pass or Fail.” Are these two the only outcomes we “should” expect when discussing executing tests? What about “Execution Abandoned”?

A new result parameter that a tester and organization can introduce to point out that execution did not happen due to some cases. It will help us peek one layer deeper without debugging. Otherwise, first, we would check why a test case failed and then notice that the execution was abandoned due to some reasons.

Such tweaks in the test execution phases enhance the analysis part and our observations angles and provide more profound meaning to test execution results, processes, and reports.

TIn general, test execution techniques can mean working on three different areas.

• Test execution states
• Test execution priorities
• Test execution phases

Let’s explore them individually in the next section of this test execution techniques tutorial.

A test execution state is commonly referred to as the current state of a test case. If a test case has passed, its test execution is labeled as “Pass” and reported in the final report. Similarly, other states may be exhibiting a different state, and since each test case has its unique test execution state, we need to display each in the final reports.

Conventionally, organizations settled on two states - “Pass” and “Fail”- a scenario similar to the example quoted in the previous section. But these two states often represent the final outcomes, not the current status.

For instance, what about the test cases that have not been executed or are under execution? They can neither be labeled as “Pass” nor “Fail.” Due to this, today, organizations divide test execution states into multiple logical divisions that help analyze the test cases more clearly.

The following table displays an example of such divisions. Although, please note that here we use the word “example” because there are no such strict protocols or standards regarding test execution states. It is a technique that depends on the organizations and testers completely. They can define their states with their meanings. However, the representation should be done in a table like the one given below.

These test execution states may require help from the tool, too, so that you can input various conditions for various states. Therefore, test planning is necessary, and test execution techniques should always be a part of it.

Prioritization has been a method adopted deep into each domain of computers and the science behind it. We even prioritize the jobs that are received by the CPU for execution. A similar thing can be adopted in the test execution technique to divide test cases according to priority and find defects in high-priority cases first.

Bringing prioritization into test execution can help us in many ways. A few can be:

• Test execution priority system executes high-priority test cases first. With this approach, we can identify critical bugs first rather than waiting for the execution to be completed, which generally takes hours (sometimes days).
• If the project is running out of deadline and we need to cut a few test cases, we don’t need to go through the complete suite. All we need is to delete the lowest priority cases.
• As the software scales, the test case number grows along with it. At a point, it becomes impossible to test each line of code. During such times, we rely on prioritization to assemble and construct high-priority cases and gradually move down the ladder as time allows.

Now comes the central part, how do we prioritize test cases? How do we classify a test case to be of high priority? More importantly, what kind of priority are we talking about here?

Before we move on to the actual classification, we need to set the priorities. Similar to other sections, here, too, everything depends on testers, organization, and the project. You can develop your classifications and models based on the type of project you are dealing with. However, considering a general consensus, an organization may pick one of the following test execution priorities.

A risk-based prioritization of test execution would mean that a high-priority test case is associated with high risk. Simply put, it is a high priority if a test case fails and can lead to critical issues such as functionality not working. Subsequently, when we deal with test cases that may be extremely rare and associated with a very small part of the application, it is of a lower priority.

A problem in risk-based prioritization is defining the risks of the system and correlating them with test cases to assign priority. Popularly, we follow two procedures here.

The first approach focuses on listing problems that could affect the application badly. Then, we take a test case and define the probability of occurrence of each issue if that case is not executed. The higher the likelihood of a major issue, the higher the test case takes priority.

The second approach talks about listing the risks by referring to the requirements satisfied by the software. Next, we calculate the risk exposure of the requirements. Then, we calculate the risk exposure of the items, and based on that; we prioritize each test case.

However, please keep in mind that there are no strict protocols here. If the software project is unsatisfied with the above methods, a tester can devise their own.

As the name suggests, coverage-based priority refers to assigning a higher priority to the test case that covers more code. Sometimes testers use branch coverage instead of code coverage which is completely a personal choice to use. However, please refrain from using line coverage. If priority is done based on how many lines are covered, sometimes, you may miss critical branch test cases that should have been at the top spot.

Another test execution priority technique is the history-based prioritization of test cases. As its name suggests, it is a straightforward method of assigning priority to a test case based on its historical data.

In history-based test case prioritization, we analyze the test case interaction with the test cases, and if it has failed, we assign it more priority. The ideology behind this process is that if a regression run in the past has been executed, then the code changes must be made by keeping in mind the failed test cases. If passing them is the priority, they should be executed logically.

Here, we work along these steps.

• First, we calculate the number of test cases that cover each function. Then we weight each function such that its weight has an inverse exponential relationship to the number of test cases that covered it and such that the total weights over all functions sum to 1.
• Next, we define the test history. This is the sum of the weights of all functions that the test case covered.

This is used as a criterion to define the next run's priority directly.

The most simple type of test execution priority is requirement based. Here, we simply refer to the requirement document and compare each test case with the given requirements. If the requirement is of a higher priority, we prioritize the test case; else, the test case is given a lower priority.

If the tester and the organization are clear about these two factors, they can proceed with this much information. However, since it is generally not the case, testers use one of the following four methods to analyze the priorities.

• Customer Assigned Priority (CAP): This method refers to considering the significance of a requirement from a customer’s point of view.
• Requirement Volatility (RV): It considers the significance of a requirement based on how frequently it has changed.
• Implementation Complexity (IC): It involves considering the significance of a requirement based on how much time it took for its construction.
• Fault Proneness (FP): In this method, one can consider the significance of a requirement based on how much failure it has taken during development. This includes field failures along with development failures.

Instead of going ahead with a single approach, testers can (and should) combine these approaches to generate an average priority value.

A simple illustration of requirement-based priority is shown below.

In the above image, the “Prioritization Factor Value” method is computed based on each of the four classifications of requirement-based prioritization. Based on it, a weighted priority is calculated, which is the final priority of the test case.

Challenges with Test execution priorities

Test execution priority is a part of the test execution technique, helping us evolve the methods of test execution for better efficiency. It is a proven method and is always recommended to use. However, a tester may face two small challenges in their journey of test execution prioritization.

• The first challenge is related to time investment. A one-time investment towards prioritizing the test case according to the test case may extend things back further. But this is done only once, and the priority remains the same (or almost the same) in the releases ahead.
• The second challenge is related to the changes that occur over time in any domain. When a test case is changed, it needs to be scrutinized to see if it is still following the exact requirements so we can adjust its priority. When a requirement changes from the customer's or organization’s end, we need to change all the test cases and their priorities. It becomes an additional task that consumes a lot of time.

Finally, focusing on the last part of the test execution techniques, we consider the phases in which we can divide the test execution.

While it may offer several advantages, the best one among them seems to be the careful organization it provides so that instead of considering the large picture altogether, we consider smaller units and improve them individually. This may sound similar to unit testing, and in a way, it is.

The team should create and divide phases as per their expertise and confidence. However, to demonstrate and define the most popular division of all, you may go ahead with these three phases.

Collection and Exploration phase

The first phase is the most critical as the testing quality depends on it. If you are thinking about a defect and quickly writing the script, this method strictly prohibits this process. Collection and exploration ask the tester to move in two steps. In the first step, collect the defects in one place. In the second step, start exploring the application manually and document two things - the things you think are essential to test and the defects you can uncover during this phase.

Give this phase some time, as the more a tester focuses on it, the better effect it has in the next phase.

Action phase

The second phase is the “action” phase, where we act upon all the things gathered one step above. We arrange all the findings, write scripts, prioritize them according to the method we follow, and execute them on the application.

Analysis phase

The final phase is analyzing our results and comparing what we want to achieve to reality. For instance, have we written the tests that fail? Or asserting whether the requirement we thought a test case would suffice is doing that or not.

A tester can also break down these three phases into more sub-parts or change them at your convenience. The final motive of focusing on smaller units at a time for the overall improvement of testing is the only thing that matters.

In today's increasingly competitive environment, development teams must deliver quality software faster. The popularity of agile methods makes balancing the two easier. By choosing the right automation testing platforms and following parallel testing techniques, bottlenecks can be alleviated by cutting test execution times and allowing teams to fix bugs faster.

By leveraging test orchestration and execution platforms like LambdaTest, organizations and enterprises can hasten their testing efforts and improve their product quality.

LambdaTest is a cloud-based testing platform that allows you to perform manual and automated testing for web and mobile applications. Devs and testers can quickly test their applications on a real device cloud in real user scenarios.

LambdaTest’s continuous quality cloud also provides an end-to-end test orchestration cloud - HyperExecute, which is a next-gen intelligent testing platform to help you achieve blazing fast test execution up to 70% faster than any conventional cloud grids.

Subscribe to the LambdaTest YouTube channel for test automation tutorials around Selenium, Playwright, Appium, and more.

This section marks the end of our discussion on three test execution techniques and their importance in test execution. In retrospect, as a tester, you should always remember that test execution is not a standard followed by strict protocols.

Since it is a process, it demands tweaks and slight changes over time. A tester, therefore, should always mold this process according to the comfort and benefit of the application.

While this is the representation and groundwork, the test execution phase helps us divide our work to keep us focused on small units to improve efficiency.