Appium Visual Testing: A Step-By-Step Guide

OVERVIEW

Mobile applications have drastically evolved since the very first smartphone, which was launched in 1994. This evolution has transformed how people use and interact with their devices, providing businesses new and innovative opportunities to engage with their customers.

Many businesses have also developed mobile versions of their existing web application, thus providing more opportunities to keep their customers engaged on their platforms. Due to this increase in mobile application demands, it is also very important to ensure that the applications are visually appealing and user-friendly.

Visual testing is one of the essential software testing techniques which helps evaluate the visual aspects of the application, thus ensuring that it meets the design and usability requirements. With the rise of mobile applications, visual testing has become even more important because it will ensure a smooth and seamless experience for the users. Visual testing will also ensure that the application is compatible with different devices, screen sizes, and operating systems by verifying the visual aspect of the application, such as element alignments, colors, fonts, etc.

When comparing the identification of issues in visual testing with automated functional testing, it becomes apparent that the majority of these issues cannot be automated using conventional automation tools. Even if you try to implement the verification of colors, fonts, and element alignments using a functional automation approach, it will be very complicated.

The optimal method for implementing visual testing involves utilizing tools that facilitate the comparison of current application screen screenshots with baseline screenshots. These baseline screenshots should adhere to the designated designs and possess correct element alignments. So whenever there's a new version developed, you can simply run your visual tests against that new build and compare it with the previous application version.

Visual testing can be performed on both web and mobile applications. However, for this Appium Visual Testing tutorial, we are focusing on mobile visual testing.

Benefits of Visual Testing

Before moving ahead in this Appium visual testing, let’s understand how visual testing can be beneficial to the overall quality of the application:

• Improved user experience

Visual testing ensures that the application meets the design and usability requirements which helps improve the quality of the application and thus provide a seamless experience to the users. This is done by examining the attributes of the application elements visible to the users.

• Faster feedback

Visual testing helps identify the issues much faster in the testing process, thus resulting in faster production and maintaining the quality of the application.

• Standard compliance

Visual testing helps ensure compliance with regulatory standards, branding guidelines, and user expectations. It helps identify issues that may violate standards, such as accessibility or usability.

• Cross-platform testing

Visual testing supports cross-platform testing, thus ensuring that the application works consistently across different devices, operating systems, and screen sizes.

• Cost-effective

Visual testing helps reduce the overall cost incurred by the organization by identifying the issues early in the development phase. With this early feedback on the defects, it helps reduce the extensive manual testing, thus reducing the testing cost.

Note : Automate visual testing on the Appium cloud of 3000+ real devices. Try LambdaTest Now!

What is Appium?

Appium is a freely available mobile app testing framework encompassing Android, iOS, Windows, and Mac desktop applications of various types, such as Native, Web, and Hybrid.

In the recent Appium version 2.x.x, several significant enhancements have been made to improve its overall functionality. Notable changes include the separation of drivers from the command line installation, allowing greater flexibility. Additionally, the introduction of plugins enables users to override, modify, extend, and incorporate new functionality into Appium.

Furthermore, Appium promotes the adoption of the W3C protocol for automation, replacing the previous mobile JSON wire protocol.

Getting Started with Appium Visual Testing

Apart from allowing to automate different applications, it also allows performing visual testing for mobile applications by exposing methods that help in comparing baseline screenshots with current page screenshots. However, this feature of Appium visual testing is very basic but still good enough to identify the visual differences.

Machine Setup

To get started with Appium visual testing, first, let’s see how to set up Appium on your machine, along with the tools and libraries needed to enable Appium to perform visual testing. For this Appium visual testing, I will be using a macOS machine.

Install Node.js

To install Node.js v16 using Node Version Manager (NVM), simply execute the provided command in your machine's terminal or command prompt.

> curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

When you execute this command, you will see the output as shown below:

By executing this command, the installation script for NVM will be downloaded and subsequently run on your machine to facilitate the installation process.

After successfully installing NVM, proceed to install Node.js 16.x.x LTS by executing the given command.

> nvm install lts/gallium

After the installation is complete, you can verify the installed Node version by executing the following command:

Pre-requisites

To ensure successful compilation and execution of your tests, it is necessary to download and install the following components:

JDK 11: To obtain the JDK 11 installer, visit the Eclipse Temurin site and choose the appropriate installer for your operating system. Alternatively, if you are using a macOS machine, you can utilize Homebrew to install the JDK by executing the provided command.

> brew tap homebrew/cask-versions
> brew install --cask temurin11

In this context, we opt not to install Oracle JDK due to its licensing limitations, as it requires a paid commercial license for professional use. Additionally, we are avoiding the use of OpenJDK as it is discouraged on the Which JDK site. Instead, it is highly recommended to utilizeTemurin JDK, as it is endorsed on the same site, and that is the JDK we are utilizing in this scenario.

Maven: To install the distribution package, follow the provided link, unzip it in a desired location, and configure the M2_HOME environment variable. Alternatively, on macOS, you can effortlessly install Maven by executing the command "brew install maven".

Install Android Studio

To install the most recent version of Android Studio, navigate to the Android download page, download the installer, and initiate its execution. After completing the installation, launch the SDK Manager and install the specified packages, as demonstrated in the accompanying screenshot below.

Take note of the package enclosed in the red box, which will become visible only if you deselect the option to Hide Obsolete Packages.

In the final step, you will need to configure the ANDROID_HOME environment variable to indicate the SDK path. Additionally, it is necessary to update the PATH variable with the paths to the emulator, tools, platform-tools, and cmdline-tools.

export ANDROID_HOME="/path/to/Android/sdk"


export PATH=$PATH:$ANDROID_HOME/emulator
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME/tools/bin
export PATH=$PATH:$ANDROID_HOME/platform-tools
export PATH=$PATH:$ANDROID_HOME/cmdline-tools/latest/bin

Install Appium Server CLI

To automate Android apps on your machine using the Appium server, installing the most recent version of the Appium Command Line tool for managing the server and drivers is essential. To install the Appium CLI, execute the following command in your terminal:

The installation process will exclusively include the latest Appium v2.0.0 CLI on your machine. To verify whether the installation of Appium v2.0.0 was successful, execute the following command:

Get Available Appium Drivers

In Appium 2.0, the drivers have been separated from the Appium CLI installation package. Consequently, installing the Appium CLI alone will not include any drivers. To view the complete list of drivers supported by Appium, execute the following command in your terminal:

Install Appium Driver

Based on your specific needs, you can install any desired driver from the provided list of supported drivers. For instance, if you wish to install the uiautomator2 driver for Android, execute the following command:

To verify the successful installation of the driver, you can once again execute the previously mentioned command to list all the supported drivers. If the driver has been installed correctly, you will notice a tick (✔️) displayed next to it.

Install Appium Doctor

Within the Appium ecosystem, there exists the Appium Doctor CLI tool, which serves to validate the correctness of your machine setup. To install appium-doctor, execute the following command:

Check Machine Setup

After successfully installing Appium Doctor on your machine, execute the following command in the terminal to verify if the machine setup is correctly configured for automating Android apps:

Install OpenCV

OpenCV is a library required by Appium to perform visual testing by screenshot comparisons.

For macOS

Install the OpenCV core library on your Mac machine by running brew install opencv.

For Windows

Run choco install OpenCV .on your Windows machine

For Linux

On a Linux machine, you must build OpenCV from the source manually.

Once the core OpenCV is installed, you must create the OPENCV_LIB_DIR .environment variable with the path where the OpenCV library was installed.

Next, you must install its Node library by running the following command:

As you can see, we are using the OPENCV4NODEJS_DISABLE_AUTOBUILD environment variable while executing this command. It is required to disable the auto-build of the library during installation since we have already installed the core library in an earlier step.

Install Images Appium Plugin

With the latest Appium version 2.x.x, the Appium team has moved the image comparison-related method implementations to a separate plugin called the images plugin, which you must install before you start Appium visual testing. To install this plugin, run the following command:

If this plugin is not installed, you will encounter an error like Method not implemented when you call the method, which helps in Appium visual testing.

How to Write Visual Tests with Appium?

For this Appium visual testing tutorial, we will be using a couple of versions of the Proverbial application (Proverbial New and Proverbial Old ) for the Android platform, and both versions will have completely different home pages. So when we write the test, first, we will create a baseline image using the older version of the application. When testing the newer version of the application, we will compare its home page screenshot with the baseline image we saved earlier.

In this tutorial on Appium visual testing, the code examples which will be demonstrated are available on GitHub. Feel free to clone the repository and follow along. Let’s take a look at the code examples.

Setup Project Dependencies

We will create a Maven project and set up the pom.xml file as demonstrated below:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://maven.apache.org/POM/4.0.0"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
   <modelVersion>4.0.0</modelVersion>


   <groupId>com.github.wasiqb.appium</groupId>
   <artifactId>appium-visual-regression-sample</artifactId>
   <version>1.0-SNAPSHOT</version>


   <properties>
       <maven.compiler.source>11</maven.compiler.source>
       <maven.compiler.target>11</maven.compiler.target>
       <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
   </properties>


   <dependencies>
       <dependency>
           <groupId>org.seleniumhq.selenium</groupId>
           <artifactId>selenium-java</artifactId>
           <version>4.8.1</version>
       </dependency>
       <dependency>
           <groupId>io.appium</groupId>
           <artifactId>java-client</artifactId>
           <version>8.3.0</version>
       </dependency>
       <dependency>
           <groupId>org.testng</groupId>
           <artifactId>testng</artifactId>
           <version>7.7.1</version>
           <scope>test</scope>
       </dependency>
       <dependency>
           <groupId>com.google.truth</groupId>
           <artifactId>truth</artifactId>
           <version>1.1.3</version>
           <scope>test</scope>
       </dependency>
       <dependency>
           <groupId>com.google.guava</groupId>
           <artifactId>guava</artifactId>
           <version>31.1-jre</version>
       </dependency>
   </dependencies>
</project>

Here, we are using the latest version of Appium which will be used for visual testing our Android application.

Start Appium Server

First, we must start the Appium server from our test. This can be achieved by using the following code snippet:

private static final String USER_DIR = getProperty ("user.dir");


private AppiumDriverLocalService buildAppiumService () {
   final var logFile = Path.of (USER_DIR, "logs", "appium.log")
       .toFile ();
   final var builder = new AppiumServiceBuilder ();
   return builder.withIPAddress (System.getProperty ("host", "127.0.0.1"))
       .usingPort (Integer.parseInt (System.getProperty ("port", "4723")))
       .withLogFile (logFile)
       .withArgument (GeneralServerFlag.BASEPATH, "/wd/hub")
       .withArgument (GeneralServerFlag.USE_DRIVERS, "uiautomator2")
       .withArgument (GeneralServerFlag.USE_PLUGINS, "all")
       .withArgument (GeneralServerFlag.SESSION_OVERRIDE)
       .withArgument (GeneralServerFlag.LOCAL_TIMEZONE)
       .build ();
}

Here you can see, we are setting:

• IP Address: We are checking if the host name is provided as a system property if it's not provided, then we will use 127.0.0.1 as the default host IP address.
• Port: We are checking if the port is provided as a system property. If it’s not provided, we will use 4723 as the default.
• Log File: Here, we are setting the file path where we want to save all the Appium logs.
• Base Path: We are setting the base path for the Appium server as /wd/hub.
• Use Drivers: We are using the uiautomator2, which is Android specific
• Use Plugins: We need to enable the images plugin, which we can enable by passing all to enable all the installed plugins. We are not passing images here because the Appium Java client currently gives an error when images are passed as an argument.
• Session Override: We have enabled session override for the test we will be running. This setting will help override the already running session on the same host and port.
• Local Timezone: We have also enabled logging with local timezone timestamps.

Setup Android Driver

Next, we will create Android driver options and initialize the Android driver as demonstrated in the code snippet below:

private static final String DEVICE_NAME_KEY = "deviceName";
private static final String DEVICE_VERSION_KEY = "deviceVersion";
private static final String USER_DIR = System.getProperty ("user.dir");


private AndroidDriver driver;
private boolean isBaseline;
private AppiumDriverLocalService service;


private Capabilities buildCapabilities (final String appName) {
   final var deviceName = System.getProperty (DEVICE_NAME_KEY, "Pixel_6_Pro");
   final var deviceVersion = System.getProperty (DEVICE_VERSION_KEY, "11");
   final var options = new UiAutomator2Options ();
   options.setPlatformName ("Android")
       .setPlatformVersion (deviceVersion)
       .setDeviceName (deviceName)
       .setAvd (deviceName)
       .setApp (Path.of (USER_DIR, "src/test/resources", MessageFormat.format ("{0}.apk", appName))
           .toString ())
       .setAutoGrantPermissions (true)
       .setFullReset (true)
       .setCapability ("appium:settings[ignoreUnimportantViews]", true);
   return options;
}


@BeforeClass (alwaysRun = true)
@Parameters ({ "isBaseline", "appName" })
public void setupSuite (@Optional ("false") boolean isBaseline, String appName) {
   this.isBaseline = isBaseline;
   this.service = buildAppiumService ();
   this.service.start ();
   this.driver = new AndroidDriver (this.service.getUrl (), buildCapabilities (appName));
}


@AfterClass (alwaysRun = true)
public void teardownClass () {
   this.driver.quit ();
   this.service.stop ();
}

Code Walkthrough

Step 1

We are using UiAutomator2Options to set up all the different Android capabilities, which will install the application based on the appName provided to this method.

We also have the deviceName variable, which will get the device name from the system properties. If it is not provided, then by default, we will use Pixel_6_Pro . Similarly, we have deviceVersion , which will get the device platform version from the system properties. If it is not provided, then by default, we will use 11 as the platform version.

We build the application path using the Path class by getting the application saved in the {project-root}/src/test/resources/{app-name}.apk where the app-name depends on the application's version being tested.

Step 2

In the setupClass method, we set up our test class by initializing the Appium service and then starting the server. Once the server has been started, we initialize the Android driver with the capabilities we set earlier and use the Appium service URL to connect our test to the Appium server, which was started by our test.

Step 3

In the teardownClass method, we are quitting the driver session and stopping the running Appium server.

Performing Visual Comparison

Now, this is the most critical step where we implement visual image comparison logic to our test so we can assert and fail the test if there are any inconsistencies in the UI design and alignment, if it is more than the visual threshold, we have set in our test. Let's see how to implement visual comparison as shown below:

import static org.openqa.selenium.OutputType.FILE;
import static com.google.common.truth.Truth.assertWithMessage;


private static final double VISUAL_THRESHOLD = 0.99;
private static final String SCREEN_NAME = "Home-page";


@Test
public void testAppVisual () throws IOException {
checkVisual ();
}


private void checkVisual () throws IOException {
final var baseImage = getBaseLineImage ();
final var actualImage = getActualImage ();
final var diffImage = getDiffImage (actualImage);
final var score = getVisualScore (baseImage, actualImage, diffImage);


assertWithMessage ("Visual result").that (score.getScore ())
.isAtLeast (VISUAL_THRESHOLD);
}


private File getActualImage () throws IOException {
final var actualImage = this.driver.getScreenshotAs (FILE);
FileUtils.copyFile (actualImage, getFile ("actual"));
return actualImage;
}


private File getBaseLineImage () throws IOException {
final var baseImage = getFile ("baseline");
if (isBaseline || !baseImage.exists ()) {
final var newBaseline = this.driver.getScreenshotAs (FILE);
FileUtils.copyFile (newBaseline, baseImage);
}
return baseImage;
}


private File getDiffImage (final File actualImage) throws IOException {
final var diffImage = getFile ("diff");
FileUtils.copyFile (actualImage, diffImage);
return diffImage;
}


private File getFile (final String fileType) {
return Path.of (USER_DIR, "images", fileType, MessageFormat.format ("{0}-similarity.png", SCREEN_NAME))
.toFile ();
}


private SimilarityMatchingResult getVisualScore (final File baseImage, final File actualImage, final File diffImage)
throws IOException {
final var options = new SimilarityMatchingOptions ();
options.withEnabledVisualization ();


final var res = this.driver.getImagesSimilarity (baseImage, actualImage, options);
res.storeVisualization (diffImage);


return res;
}

Code Walkthrough

Step 1

First, there is the VISUAL_THRESHOLD constant declared, which has the minimum threshold of 99% of the differences, which can be considered fine when performing Appium visual testing. Anything less will fail the test because it will indicate more differences in the actual image when compared with the baseline image.

Next, we will use the SCREEN_NAME constant to store the screen name, which will be visually tested.

Step 2

Step 3

In the checkVisual method, baseline image, actual image, and difference image are being fetched. Then, the comparison is made using both the baseline and actual image, and the matching score is derived. That score is asserted against the threshold, which is set as 0.99 to make sure that the score is at least equal to the threshold; it should not be less than 0.99.

Step 4

In the getBaseLineImage method, we check if the getBaseLineImage flag is true or not. If it is true then the baseline image is captured and saved. We also check if there is any baseline image in the first place, irrespective of the isBaseline flag.

Step 5:

In the getActualImage method, we capture the current screenshot and save it in the actual image folder.

Step 6

In the getDiffImage method, we are currently saving the actual image as it is without any modification. This is because when Appium finds the differences and tries saving to a file path that does not exist, it will give an error.

Step 7

In the getFile method, we build the file path depending on the file type provided in the parameter.

Step 8

In the getVisualScore method, we use the getImagesSimilarity method from the AndroidDriver instance. This method performs P2P (Pixel to Pixel) comparison while comparing the actual image with the baseline image.

We are also saving the difference image on the path where we had created a placeholder image earlier by using the storeVisualization method.

Execution of Visual Tests

Now, we will create a TestNG XML file which will help in performing Appium visual testing. Following is the content of the file:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Appium Visual Testing Suite" verbose="2">
<test name="LambdaTest Appium Baseline Test">
<parameter name="isBaseline" value="true"/>
<parameter name="appName" value="proverbial_old"/>
<classes>
<class name="com.github.wasiqb.appium.VisualSimilarityTest"/>
</classes>
</test>
<test name="LambdaTest Appium Actual Test">
<parameter name="appName" value="proverbial_new"/>
<classes>
<class name="com.github.wasiqb.appium.VisualSimilarityTest"/>
</classes>
</test>
</suite>

As you can see, there are a couple of parameters being declared in the testng.xml file, which are:

• isBaseline: This is an optional parameter, if set to true means it will force updating of baseline images, else existing baseline images will be used. Its default value is false
• appName: This parameter will have the application name, which will be used to install on the device.

We have two tests, one of which is using an old proverbial application that is considered as stable. Hence, we are creating the baseline image using that application. The second application is the updated version of the same application where the home screen is now modified. This application will be visually tested with the old one.

Let’s see how each of these applications' home screens look. Following is the home screen from the old application:

And this is how the new application home screen looks like:

When we execute this test suite, the first test will generate the baseline image while the second test will compare the new application screen with the baseline and fail the test if there are any differences between both the screens more than the set threshold. Following is the difference image generated by Appium:

Limitations of Appium Visual Testing

As you can see, Appium easily highlights the difference between the old version and the new version of the applications without any problems. However, there are a few limitations when it comes to Appium visual testing. Let’s check them out.

• Ignore Dynamic element

n Appium, you cannot ignore the dynamic area on the screen. Suppose there is some area on the screen that might change every time we run the test. Ideally, that element area should not be verified. These features of ignoring cannot be done with Appium.

• Maintaining Baseline images

Another thing to note here is that it is very difficult to maintain the baseline images. In our example, it was only one page that we verified, but in a real-world project, there can be hundreds of pages that will be verified, and maintaining that many baseline images is not easy.

• Update Baseline Images

Also, to update the baseline image in case there are any design changes in the application, with Appium, you need to do that manually by copy-pasting the actual image to the baseline image folder.

• Visual Dashboard View

Another limitation is the lack of a dashboard view to view all the failed test cases, and for each test case, you could see the baseline and actual image side-by-side and an option to highlight the differences with a click of a button.

Visual Testing with Smart UI

Until now, we have seen how to perform visual testing using vanilla Appium. We also understood the limitations when performing Appium visual testing. Now, some cloud-based platforms provide Visual testing support and reduce most of the limitations we have seen earlier, thus providing users with a robust visual testing solution. One such cloud platform is LambdaTest. It is a digital experience testing platform that offers manual and automated testing for web and mobile applications. It has a vast range of browsers and devices and provides support to perform Visual testing for Web and Mobile devices.

The visual testing feature provided by LambdaTest is called Smart UI. This feature is available as a separate section on the LambdaTest dashboard. It allows you to customize visual thresholds, ignore specific regions from the screenshots, and a complete dashboard where you can view the test results and accept the images or raise a bug.

Visit our documentation to get started with Visual testing using Appium.

You can also subscribe to the LambdaTest YouTube Channel and stay updated with the latest tutorial around Selenium testing, Playwright, Appium, and more.

While performing Appium visual testing, one thing to keep in mind is it is possible that there can be a negligible amount of difference which can occur due to minor differences in the time visible on the screen when comparing with the baseline image, or it can be due to dynamic text element where the text can change on each test execution, for example, price of the product, or any other dynamic text. To ignore such differences, you can adjust the threshold value in Smart UI, which we will see later in this Appium visual testing tutorial.

Let’s convert the test we had written with vanilla Appium earlier to a Smart UI test.

Upload Applications on the Cloud

The first step is to upload all the different versions of the applications on the LambdaTest cloud. On the LambdaTest Dashboard, navigate to Real Device - App Automation and click on the Upload App button towards the right of the screen.

When you click on this button, you will be able to see the following screen:

On this page, you can select the .apk file and upload it to the cloud servers. Once the application is uploaded successfully, you will get a unique application URL which you must copy and save as an environment variable named LT_APP.

Create Smart UI Project

On the LambdaTest Dashboard, click on the Smart UI link on the left side navbar. Once the screen opens, click the New Project button on the top right of the screen. You will see the following screen:

Since we are performing visual testing on mobile devices, you must select the Real Device option, give a unique project name, select the Approvers for the project, and mention tags that can be used for better filtering on the dashboard.

Setup Android Driver

Next, we will set up an Android driver by updating the capabilities specific for LambdaTest as demonstrated below:

private static final String DEVICE_NAME_KEY = "deviceName";
private static final String DEVICE_VERSION_KEY = "deviceVersion";


private String appUrl;
private String buildName;
private AndroidDriver driver;
private boolean isBaseline;


@BeforeClass (alwaysRun = true)
@Parameters ({ "isBaseline", "buildName", "appUrl" })
public void setupClass (@Optional ("false")
final boolean isBaseline, final String buildName, final String appUrl)
throws MalformedURLException {
this.isBaseline = isBaseline;
this.buildName = buildName;
this.appUrl = appUrl;
final var userName = System.getenv ("LT_USERNAME");
final var accessKey = System.getenv ("LT_ACCESS_KEY");
this.driver = new AndroidDriver (
new URL (MessageFormat.format ("https://{0}:{1}@mobile-hub.lambdatest.com/wd/hub", userName, accessKey)),
buildCapabilities ());
}


@AfterClass (alwaysRun = true)
public void teardownClass () {
this.driver.quit ();
}


private Capabilities buildCapabilities () {
final var deviceName = System.getProperty (DEVICE_NAME_KEY, "Pixel 5");
final var deviceVersion = System.getProperty (DEVICE_VERSION_KEY, "11");
final var options = new UiAutomator2Options ();


final var ltOptions = new HashMap<String, Object> ();
ltOptions.put ("w3c", true);
ltOptions.put ("platformName", "Android");
ltOptions.put ("deviceName", deviceName);
ltOptions.put ("platformVersion", deviceVersion);
ltOptions.put ("app", this.appUrl);
ltOptions.put ("devicelog", true);
ltOptions.put ("visual", true);
ltOptions.put ("network", true);
ltOptions.put ("video", true);
ltOptions.put ("build", "Sample Build");
ltOptions.put ("name", "Sample Test");
ltOptions.put ("project", "Sample Project");
ltOptions.put ("isRealMobile", true);
ltOptions.put ("autoGrantPermissions", true);
ltOptions.put ("smartUI.project", "Sample Visual Regression");
ltOptions.put ("smartUI.build", this.buildName);
ltOptions.put ("smartUI.baseline", this.isBaseline);


options.setCapability ("lt:options", ltOptions);


return options;
}

Here, we first get the credentials for LambdaTest from the environment variables like LT_USERNAME, which will have LambdaTest username, LT_ACCESS_KEY which will have the access key. These credentials are available on the dashboard as shown below:

Next, all LambdaTest related capabilities are set. However, there are few capabilities, which we are updating dynamically via system properties/environment variables. Let’s understand those capabilities:

• Device Name: Device name is initially fetched from system properties. If it is not available, then by default Pixel 5 is used as the device.
• Device Version: Device version is fetched from system properties. If it is not passed there, then by default, 11 is used.
• Build Name: Build name is being passed as TestNG parameters. This build name is to be used to set Smart UI build for the mentioned Smart UI project. The baseline image will be saved in one build and the actual image comparison will be done in another build.
• Baseline: This is a boolean flag, which is firstly fetched from the TestNG parameter, if it is not provided, then by default, false is used.
• Application: We get the LambdaTest application URL from the TestNG parameter. We will pass an old application URL for the first test where we will create the baseline image, and later we will pass the new application URL for the second test where we will validate the new application with the old application.

Lastly we are quitting the driver session in the teardownSuite method.

Perform Visual Comparison

Now, we will write a single test, which will be executed for old as well as new applications. Let’s see what this test class looks like as shown below:

package com.github.wasiqb.appium;


import java.net.MalformedURLException;
import java.net.URL;
import java.text.MessageFormat;
import java.util.HashMap;


import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.android.options.UiAutomator2Options;
import org.openqa.selenium.Capabilities;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Optional;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;


public class LTVisualSampleTest {
private static final String DEVICE_NAME_KEY    = "deviceName";
private static final String DEVICE_VERSION_KEY = "deviceVersion";
private static final String SCREEN_NAME        = "Home-page";


private AndroidDriver driver;


@BeforeClass (alwaysRun = true)
@Parameters ({ "isBaseline", "buildName", "appUrl" })
public void setupClass (@Optional ("false") final boolean isBaseline, final String buildName, final String appUrl)
throws MalformedURLException {
final var userName = System.getenv ("LT_USERNAME");
final var accessKey = System.getenv ("LT_ACCESS_KEY");
this.driver = new AndroidDriver (
new URL (MessageFormat.format ("https://{0}:{1}@mobile-hub.lambdatest.com/wd/hub", userName, accessKey)),
buildCapabilities (isBaseline, buildName, appUrl));
}


@AfterClass (alwaysRun = true)
public void teardownClass () {
this.driver.quit ();
}


@Test
public void testAppVisual () {
checkVisual ();
}


private Capabilities buildCapabilities (final boolean isBaseline, final String buildName, final String appUrl) {
final var deviceName = System.getProperty (DEVICE_NAME_KEY, "Pixel 5");
final var deviceVersion = System.getProperty (DEVICE_VERSION_KEY, "11");
final var options = new UiAutomator2Options ();


final var ltOptions = new HashMap<String, Object> ();
ltOptions.put ("w3c", true);
ltOptions.put ("platformName", "Android");
ltOptions.put ("deviceName", deviceName);
ltOptions.put ("platformVersion", deviceVersion);
ltOptions.put ("app", appUrl);
ltOptions.put ("devicelog", true);
ltOptions.put ("visual", true);
ltOptions.put ("network", true);
ltOptions.put ("video", true);
ltOptions.put ("build", "Sample Build");
ltOptions.put ("name", "Sample Test");
ltOptions.put ("project", "Sample Project");
ltOptions.put ("isRealMobile", true);
ltOptions.put ("autoGrantPermissions", true);
ltOptions.put ("smartUI.project", "Sample Visual Regression");
ltOptions.put ("smartUI.build", buildName);
ltOptions.put ("smartUI.baseline", isBaseline);


var smartOptions = new HashMap<String, Object> ();
smartOptions.put ("largeImageThreshold", 1200);


ltOptions.put ("smartUI.options", smartOptions);


options.setCapability ("lt:options", ltOptions);


return options;
}


private void checkVisual () {
this.driver.executeScript (MessageFormat.format ("smartui.takeScreenshot={0}", SCREEN_NAME));
}
}

Code Walkthrough

Step 1

We have a single test which will simply call the method checkVisual method. If you see in that method, we are executing a custom Smart UI script to take the screenshot of the screen with the provided screen name.

Step 2

Next, the LambdaTest Smart UI specific capabilities are setup along with other LambdaTest capabilities which we have seen about in the previous section.

Step 3

Lastly we create the driver instance using the capabilities we have built in the last step.

Executing the Tests

Now, let's see how to execute these tests using TestNG. First, we must create a testng-lt-appium.xml file with the following contents:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Appium Visual Testing with LambdaTest Suite" verbose="2">
<test name="LambdaTest Smart UI Baseline Test">
<parameter name="isBaseline" value="true"/>
<parameter name="buildName" value="Test 10"/>
<parameter name="appUrl" value="<old-app-url>"/>
<classes>
<class name="com.github.wasiqb.appium.LTVisualSampleTest"/>
</classes>
</test>
<test name="LambdaTest Smart UI Actual Test">
<parameter name="isBaseline" value="false"/>
<parameter name="buildName" value="Test 11"/>
<parameter name="appUrl" value="<new-app-url>"/>
<classes>
<class name="com.github.wasiqb.appium.LTVisualSampleTest"/>
</classes>
</test>
</suite>

When you execute this TestNG file, the first test will create the baseline images build named Test 10 using the old stable application, and the next test will run the visual test on the build named Test 11 with a new application version and check against the baseline build.

Smart UI Result View

Once your tests are executed, you will be able to see the test results for your new application on the LambdaTest Smart UI window, as shown below:

As you can see on this screen, you will see an image comparison of the new application with the baseline image of the old application. Here, you can either accept the differences and mark the test as pass or raise an issue and assign the issue to the concerned teams.

Customizations via Settings Page

You can do many different customizations on the image comparison logic when you go to the project's settings page. Here’s what different customization you can do:

As you can see on this page, you can set the following settings for image comparisons done for the project:

• Ignore Pixel Scaling: Here, you can ignore different options while doing image comparisons.
• Resize Image: You can also resize the image according to the baseline images. Transparency: You can strike balance with the baseline by updating the transparency settings.
• Error Type Identifier: You can update the settings which will be used to Error out the visual test based on these settings.
• Error Highlight Color: You can update the color you want the difference in the image to be highlighted with.
• Pixel Threshold: You can update the image pixel comparison threshold. This threshold will help in determining whether the image comparison failed or is within the threshold limit.
• Custom mix match % acceptance ratio: Here you can configure acceptance ratio which will be used to accept the image after comparing it to the baseline image. Anything beyond this ratio, will hold the image for manual investigation for the approvers to look into it.
• Bounding Boxes: This setting helps you to select the box area on the screen which you want to be either ignored or to be the area which should be selected for comparison.
• Ignore areas with color: This setting will help to ignore the area on the screen based on the colors which you specify here.

Customizations via Capabilities

Apart from the project settings page, you can also set those settings via the capabilities which are used to start the session. Let’s see an example on how to set Smart UI option using capabilities:

ltOptions.put ("smartUI.project", "Sample Visual Regression");


var smartOptions = new HashMap<String , Object> ();
smartOptions.put ("largeImageThreshold", 1200);


ltOptions.put ("smartUI.options", smartOptions);


options.setCapability ("lt:options", ltOptions);

Here, we are setting a largeImageThreshold setting which accepts a minimum of 100 and a maximum of 1200 values.

Similarly, you can also set options for the following:

• errorType: This option takes movement and flat as allowed values.
• ignore: This option takes antialiasing, alpha, colors, and nothing as allowed input values which corresponds to ignoring those attributes when executing visual tests.
• transparency: This option takes a minimum value of 0 and a maximum value of 1 with one decimal point value.
• boundingBoxes: This option takes in a list of maps containing the lengths of different sides of the box, which you want to compare with the baseline images.
• ignoredBoxes: This option takes in a list of maps containing the lengths of different sides of the box which you want to ignore from the baseline images
• ignoreAreasColoredWith: This option takes in a map of RGBA, which will represent the color that will be ignored during image comparison with the baseline image.

"Transform visual testing today with LambdaTest storybook visual testing! Elevate user experiences through effortless screenshot capture, comparison, and optimization."