Explore everything you need to know about headless browsers for testing, including benefits, testing frameworks like Selenium, and advanced techniques.
OVERVIEW
Web developers rely on popular browsers to ensure their web applications work seamlessly for users. Chrome, Firefox, Microsoft Edge, Safari, Opera, and Brave are among the most popular choices among the testers and QAs. These modern browsers come with resource-intensive graphical user interfaces (GUIs), so “Headless Browser” comes to the rescue.
A Headless Browser is a browser that executes a script without a GUI. In contrast to a real browser, a headless browser's user interface communicates with websites programmatically rather than presenting the content in a visible window. Testers and developers can automate tasks like website monitoring, web scraping, and automation testing that don't require a GUI.
The objective of this blog is to explore the significance of headless browsers, performing automation tests on well-known frameworks, their advantages and disadvantages, and best practices for efficient headless browser testing.
A headless browser is a browser without a graphical user interface(GUI), also known as “head”. Headless browsers are accessible via a command-line interface or using network connectivity. It offers automated control of a web page in an interface similar to that of common web browsers like Chrome. It is essential for web testing because it can accurately render and parse HTML, including layout, colors, fonts, and dynamic features like JavaScript and Ajax. Some popular headless browser options are PhantomJS, Headless Chrome, and Firefox in headless mode.
A Headless browser is an important tool in various fields of automation, web development, and testing due to the following:
Headless browsers are not inherently better than real browsers in every aspect. However, if we compare practically in terms of execution speed and resource usage, headless browsers will outperform real browsers. In this section, we will practically compare both browsers by running the same test case on Chrome browser using Selenium and Python.
For the test case, we will use the Lambdatest e-commerce website. We will fetch the website's title and then search for a product named iPhone.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options as ChromeOptions
import time
def scrape_table_data():
start = time.time()
# Navigate to the url
driver.get("https://ecommerce-playground.lambdatest.io/")
print(driver.title)
# Find the search input field
search_input = driver.find_element(By.NAME, "search")
# Enter "iphone" into the search bar
search_input.send_keys("iPhone")
# Submit the search form
search_input.submit()
# Close the driver
driver.quit()
end = time.time()
execution_time = (end - start) * 10**3
return execution_time
driver = webdriver.Chrome()
print("Chrome Browser")
execution_time = scrape_table_data()
print(f"Time taken: {execution_time:.03f}ms
")
options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
print("Chrome Headless Browser")
execution_time = scrape_table_data()
print(f"Time taken: {execution_time:.03f}ms")
Console Output:
As you can see there is a drastic difference of 1817.872ms for just a single test. If we create it for multiple test cases then this difference is significant. If you are confused with the code then there is no need to worry about it. In the coming blog, we will see it in detail.
Traditionally UI testing is used by developers to check the functionality of websites, but it has limitations including instability and slow execution. A paradigm shift is used to overcome these problems in headless browser testing. In contrast to UI-driven testing, it runs without loading the application's GUI and interacts directly with the website's document object model(DOM) to produce more consistent and repeatable findings. This shift has transformed testing approaches for the better.
Furthermore, since web applications change over time, headless browser testing is essential for assuring the creation of error-free and responsive web apps. It integrates smoothly with CI/CD pipelines, empowering development teams to confidently provide excellent online experiences to users. production teams may streamline their testing procedures and guarantee cross-browser compatibility by utilizing complete solutions offered by top platforms like LambdaTest, opening the door for the production of world-class web apps.
The following situations offer special uses for headless browser testing, which is a flexible and useful tool for software testing:
Modern web development frequently uses headless browser testing, and many testing frameworks support it. Here's how to utilize some well-liked testing frameworks to carry out headless browser testing:
Selenium is an open-source framework used to automate web browsers. It enables test automation of websites or web applications across various operating systems & browsers. You can set up Selenium to run headless tests using a headless browser like Firefox Headless or Headless Chrome.
Let’s write a Python script to run a Selenium test using Headless Chrome. For the test case, we will use the LambdaTest E-commerce website.
This test case fetches the title of the website:
# Import the necessary libraries
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
# Create a ChromeOptions object
options = ChromeOptions()
# Enable the headless mode
options.add_argument("--headless=new")
# Create a ChromeDriver object
driver = webdriver.Chrome(options=options)
# Get the Ecommerce Playground website
driver.get('https://ecommerce-playground.lambdatest.io/')
# Print the title of the page
print(driver.title)
# Check if the title of the page is "Your Store"
assert "Your Store" in driver.title, "Title doesn't match 'Your Store'"
# Quit the browser
driver.quit()
Console Output:
To enable the headless option in Chrome browser, we need to use the add_argument() method of Options. The --headless=new represents the new version of the Headless Chrome.
Selenium provides a feature to take screenshots using the save_screenshot() method. Below example will illustrate how to use the save_screenshot() method.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
driver.get('https://ecommerce-playground.lambdatest.io/')
driver.save_screenshot("example.png")
driver.quit()
Taken Screenshot:
Selenium's driver.save_screenshot() method takes a screenshot of the active browser window and saves it to the given file directory. The file path must lead to a readable file and be legitimate. PNG is the format used to store the screenshot.
You can learn more about Selenium from our detailed YouTube video.
Cypress is an end-to-end testing framework created for contemporary web applications. Instead of conventional headless browsers, Cypress performs tests directly in the browser, enabling better visibility and real-time debugging. You can run tests in headless mode by setting up Cypress to use Electron as the headless browser. Cypress's strong API for interacting with your application makes writing tests and assertions simple.
Let’s create and run a test on Cypress using Headless Chrome. For the test case, we will use the LambdaTest E-commerce website.
This test case fetches the website's title. Add this below code is spec.cy.js file:
describe("template spec", () => {
it("passes", () => {
// Open the Ecommerce Playground website in headless mode
cy.visit("https://ecommerce-playground.lambdatest.io/", {
headless: true,
});
// Verify that the page title is correct
cy.title().should("eq", "Your Store");
});
});
Over in the command log you'll see Cypress display the suite, the test, and your assertion (which should be passing in green).
Console Output:
Cypress itself provides a feature to take screenshots using the screenshot() function. Below example will illustrate how to use the screenshot() function.
describe("template spec", () => {
it("passes", () => {
cy.visit("https://ecommerce-playground.lambdatest.io/", {
chromeWebSecurity: false,
headless: true,
});
// Take a screenshot of the top-left corner of the page
cy.screenshot({ clip: { x: 0, y: 0, width: 1000, height: 660 } });
});
});
Output:
The taken screenshot will in the screenshots folder.
You can learn more about Cypress from our detailed YouTube video.
Google created the Node.js library Puppeteer to manage headless Chrome or Chromium. Puppeteer provides a high-level API for automating processes like form submission, navigation, and screenshotting. It works very well for site scraping, testing, and producing PDFs. Puppeteer can be used with a visible browser window even if its default mode is headless.
Let’s create and run a test on Puppeteer using Headless Chrome. We will use the LambdaTest Playground website for the test case.
This test case fetches the website's title.
const puppeteer = require("puppeteer");
async function main() {
// Launch a headless Chrome browser.
const browser = await puppeteer.launch({
headless: "new",
});
// Create a new Puppeteer page.
const page = await browser.newPage();
// Navigate to the LambdaTest Playground page.
await page.goto(
"https://www.lambdatest.com/selenium-playground/select-dropdown-demo"
);
// Get the page title.
const title = await page.title();
// Print the page title to the console.
console.log(title);
// Close the browser.
await browser.close();
}
// Call the main function.
main();
Console Output:
Puppeteer itself provides a feature to take screenshots using the screenshot() function. Below example will illustrate how to use the screenshot() function.
const puppeteer = require("puppeteer");
async function main() {
const browser = await puppeteer.launch({
headless: "new",
});
const page = await browser.newPage();
await page.goto(
"https://www.lambdatest.com/selenium-playground/select-dropdown-demo"
);
// Capture a screenshot of the full page
await page.screenshot({ path: "./example.png" });
await browser.close();
}
main();
Taken Screenshot:
By using page.screenshot() method a snapshot of the currently displayed page will be taken and saved as example.png in the working directory.
Check out our Puppeteer tutorial for a detailed understanding of it.
Microsoft's Playwright framework, which is relatively new, offers browser automation for Chromium, Firefox, and WebKit. The author offers both headless and headful execution techniques. It assists with cross browser testing and parallel execution. You can automate a variety of browser interactions with Playwright, and you can even test mobile browsers.
Let’s create and run a test on Playwright using Firefox in Headless mode. For the test case, we will use the To-do list web app.
By default, Playwright runs the browsers in headless mode. To see the browser UI, pass the headless=False flag while launching the browser.
This test case fetches the title of the website.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.firefox.launch(headless=False)
page = browser.new_page()
page.goto("https://lambdatest.github.io/sample-todo-app/")
print(page.title())
# Check if the title of the page is "Sample page - lambdatest.com"
assert "Sample page - lambdatest.com" in page.title(), "Title doesn't match 'Sample page - lambdatest.com'"
browser.close()
Console Output:
Playwright itself provides a feature to take screenshots using the screenshot() function. Below example will illustrate how to use the screenshot() function.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.firefox.launch()
page = browser.new_page()
page.goto("https://lambdatest.github.io/sample-todo-app/")
page.screenshot(path="example.png")
browser.close()
Taken Screenshot:
In Playwright, you can use the page.screenshot() method with the path parameter set to the file path where you wish to save the screenshot to capture the complete page. For instance, you would use the following code to take a snapshot of the full website and save it as an example.png:
You can learn more about Playwright from our detailed YouTube video.
A cross-browser end-to-end testing framework called TestCafe works without WebDriver or browser add-ons. TestCafe supports several browsers and performs tests by default in headless mode. You may test your web application on many hardware and software platforms by writing tests in JavaScript or TypeScript.
Let’s create and run a test on TestCafe using Headless Firefox. For the test case, we are going to use the LambdaTest website.
This test case fetches the title of the website.
import {fixture, test } from "testcafe";
// This fixture launches the Chrome browser and navigates to the LambdaTest website.
fixture("LambdaTest Website").page("https://www.lambdatest.com/");
// This test prints the title of the LambdaTest website to the console.
test("Print the title of the LambdaTest website", async (t) => {
// Get the title of the LambdaTest website.
const title = await Selector("title").textContent;
// Print the title to the console.
console.log(title);
});
Use the following command to run the above code:
testcafe firefox:headless test_example.js
Here the test_example.js is the filename and to run the browser in headless mode you have to add firefox:headless or chrome:headless in the command.
Console Output:
TestCafe itself provides a feature to take screenshots using the takeScreenshot() function. Below example will illustrate how to use the takeScreenshot() function.
import {fixture, test } from "testcafe";
fixture("LambdaTest Website").page("https://www.lambdatest.com/");
test("Print the title of the LambdaTest website", async (t) => {
// Take a screenshot of the entire page.
await t.takeScreenshot({ path: "./lambdatest-website.png" });
});
Taken Screenshot:
The options object passed to the t.takeScreenshot() method can be used to specify the screenshot type and the file directory where the image should be saved. The screenshot won't be saved to the disk if you don't give a file path. The snapshot will be saved as a PNG file if no screenshot type is specified.
Discover a wealth of insights about TestCafe through our comprehensive YouTube video.
Testing of Headless Browser on Cloud Grid Without worrying about setting up and maintaining your infrastructure, LambdaTest is an effective way to test your web apps across various browsers and OS systems. In this section, we will do it for the Selenium Framework.
Before running a Python test on LambdaTest, follow the simple steps.
After following the above steps, running tests on a cloud grid is easy. You just have to add a few lines to the existing code.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options as ChromeOptions
username = "YOUR USERNAME"
access_key = "YOUR ACCESS KEY"
# Desired capabilities for the LambdaTest test
lt_options = {
"browserName": "Chrome",
"browserVersion": "117.0",
"platformName": "macOS Catalina",
"project": "First Selenium Test",
"build": "E-commerce web Test",
"name": "Get Website Title Test",
"Headless": True, # This line will run the browser in headless mode
"W3c": True # to run on Selenium 4
}
remote_url = "https://{}:{}@hub.lambdatest.com/wd/hub".format(username, access_key)
browser_options = ChromeOptions()
# adding the capability to the chrome
browser_options.set_capability("LT:Options", lt_options)
# initializing remote server
driver = webdriver.Remote(command_executor=remote_url, options=browser_options)
driver.get('https://ecommerce-playground.lambdatest.io/')
print(driver.title)
driver.quit()
In the above script, add your LambdaTest credentials (Username and Access Key) in the above test script or set them in your Environment Variables, as it will help the LambdaTest run tests on your account.
Get your desired capabilities generated from the LambdaTest capabilities generator
Go to the Dashboard. Find details of your test case under Automation > Web Automation.
You can also explore other available options to get a better idea of the LambdaTest platform.
You can also learn how to use Pytest from our compressive video:
Compared with traditional browser testing techniques, headless browser testing has a few important benefits:
There are various headless browsers available in the market. We are going to discuss widely used in this section.
Headless Chrome is a headless version of the well-known Google Chrome browser. In Chrome version 59, Google formally debuted it in the year 2017. Since then, it has been actively maintained by Google, which makes it the most resource and performance-efficient. Headless JavaScript and HTML5 are only a couple of the cutting-edge web technologies that Chrome is well known for supporting. For automation testing, web scraping, and other web-related operations, it offers the same rendering engine and supports the DevTools Protocol. It is renowned for being quick and working with current web technologies.
Firefox also provides a headless mode that is similar to Headless Chrome. Version 56 of Mozilla Firefox marked the experimental debut of headless mode, improving reliability and stability over time. It gives developers who prefer Firefox's rendering engine a substitute for Headless Chrome. It helps with task automation and Firefox-specific feature and behavior testing.
Nightwatch.js, a Node.js library tailored for headless browsing, streamlines web automation. With a user-friendly design and a versatile API, it seamlessly integrates with headless Chrome and Electron. Nightwatch.js excels in automating web tasks, such as running JavaScript, filling forms, and taking screenshots, making it suitable for both simple and complex tasks. Its extensibility allows for the incorporation of unique capabilities, enhancing its adaptability for various use cases.
Learn more about Nightwatch.js from our article on Headless Browser Testing Using Nightwatch JS
HtmlUnit, a Java-based headless browser, is lauded for its speed and efficiency. It's open-source, free, and actively maintained by a robust developer community. HtmlUnit excels at handling JavaScript, simulating user interactions, and automating web app testing. It's text-based, lacking visual rendering, which sacrifices visuals but ensures exceptional speed and resource efficiency.
Zombie.js is a headless browser designed specifically for testing web applications. It's based on Node.js and is renowned for being straightforward and user-friendly. Zombie.js is a popular option for online testing in Node.js applications since it can explore websites, interact with forms, and run JavaScript.
Splash is a headless browser that specializes in web scraping. It offers an HTTP API for displaying web pages, making data extraction from websites efficient. Splash excels in handling complex rendering tasks and supports JavaScript execution. Its high efficiency makes it ideal for large-scale scraping projects, including tasks like scrolling through lengthy pages and waiting for JavaScript functions to complete. With its HTTP API, integrating Splash with other tools is straightforward. Best of all, Splash is free and open source.
TrifleJS is a headless browser that utilizes the Trident rendering engine and is ideal for Windows-based automation. It's scriptable with JavaScript and particularly valuable for tasks requiring Internet Explorer compatibility, like automating logins on IE-only websites or testing IE-specific web apps, simulating user interactions, and uncovering potential application flaws.
SimpleBrowser is a lightweight and highly customizable headless browser package for .NET projects, ideal for tasks like automation testing and web interaction. It's open source, offers strong community support, and has thorough documentation, making it a dependable choice for automating online interactions in .NET applications.
PhantomJS headless browser is ideal for web scraping, automation testing, and performance analysis. It lacks a graphical interface, making it suitable for server-side tasks. With a JavaScript API, it controls web content and browser behavior. Its strength lies in handling JavaScript-heavy web pages, making it valuable for scraping dynamic content.
Explore more about PhantomJS with our blog on How To Setup And Install PhantomJS In Python.
An open-source headless browser called SlimerJS was built on top of Firefox's Gecko rendering engine. It is made to run JavaScript, interact with websites, and offer an automated JavaScript API. It is renowned for working with capabilities exclusive to Firefox. However, one major drawback is that it has not been actively maintained since 2018.
CasperJS is a navigation and testing utility for PhantomJS (and SlimerJS). It offers a high-level API for automation testing and browser interactions. CasperJS, like PhantomJS, is no longer actively maintained, so programmers are urged to look at more recent options.
For tackling specific issues in web development and testing, advanced headless testing techniques are helpful. Here are several sophisticated strategies, including handling webpages with a lot of JavaScript and manipulating user agents:
Manipulating the user agent string that the browser delivers to the server is known as user agent manipulation. For emulating various browser types or versions, this can be helpful.
Use Cases
Implementation Techniques
JavaScript and its frameworks like React.js, Vue.js, and Next.js are key components of dynamic content and user interactions on many modern websites. The use of JavaScript handling techniques is necessary for effective headless testing of such websites.
Use Cases
Implementation Techniques
Emulating different device characteristics is essential to test a website's responsiveness and operation on mobile devices.
Use Cases
Implementation Techniques
User authentication and session management are necessities for many web applications. Authentication and cookie management are required for testing such situations.
Use Cases
Implementation Techniques
By enhancing the capabilities of headless browser testing, these cutting-edge strategies let you test JavaScript-heavy websites across various platforms and contexts while taking on challenging scenarios and simulating real-world user interactions. Comprehensive web application testing requires effectively utilizing these strategies.
Following best practices can help you optimize your headless browser testing procedure and provide effective, dependable, and maintainable testing. Here are some essential recommended practices to consider:
Headless browser testing has been beneficial to businesses in a range of industries. Here are a few examples of actual use cases demonstrating the benefits of headless browser testing:
Checkout Flow Optimization:
Reliability of Payment Gateways:
Data Privacy Compliance:
Integrity of Financial Transactions:
Content Consistency:
Performance of a Website:
This guide covered the importance of headless browsers and their advantages over regular browsers. We explored headless browser testing frameworks like Selenium, Cypress, Puppeteer, Playwright, and TestCafe for tasks like extracting website titles and taking screenshots.
We delved into the benefits of cloud-based testing, examined a wide range of headless browsers, and discussed advanced strategies such as modifying user agents, handling JavaScript-heavy sites, simulating mobile devices, and managing cookies and authentication.
Best practices for headless browser testing were highlighted to ensure dependable and resilient web apps. Real-world use cases showcased the utility of headless browser testing across different contexts.
Headless browser testing is essential for delivering high-quality web experiences in today's digital landscape. Armed with the knowledge and skills from this tutorial, you're well-prepared to tackle the challenges of contemporary web development and testing.
On this page
Author's Profile
Jainish Patel
Jainish Patel is a tech enthusiast specializing in web development. With a primary focus on React.js, React.native, and core front-end languages, He is well-equipped to tackle the ever-evolving world of web technologies. Building sleek and interactive Mobile and web applications is one of his strengths. Moreover, he is passionate about crafting seamless user experiences and transforming ideas into reality through programming. You can also follow him on Twitter.
Reviewer's Profile
Shahzeb Hoda
Shahzeb currently holds the position of Senior Product Marketing Manager at LambdaTest and brings a wealth of experience spanning over a decade in Quality Engineering, Security, and E-Learning domains. Over the course of his 3-year tenure at LambdaTest, he actively contributes to the review process of blogs, learning hubs, and product updates. With a Master's degree (M.Tech) in Computer Science and a seasoned expert in the technology domain, he possesses extensive knowledge spanning diverse areas of web development and software testing, including automation testing, DevOps, continuous testing, and beyond.
Get 100 minutes of automation test minutes FREE!!