LambdaTest Helps Emburse Reduce Cost By 50% With 20% Reduction In Test Execution Time

Alan Harwood, Senior Manager, Engineering at Emburse

Headquarters: Los Angeles, California • Employees: 850+

Download PDF
Emburse

20%

Reduced Test Execution Time

50%

Reduced Cost

Emburse is a modern spend management company offering solutions that revolutionize the way organizations manage employee expenses, process invoices, and release payments. Emburse humanizes work by empowering business travelers, finance professionals, and CFOs to eliminate manual and mundane time-consuming tasks. This helps them focus on tasks that matter the most to the organization.

The organization has been working with an extensive range of customers, ranging from early-stage startups to multinational global organizations with hundreds of thousands of employees, accounting to 16,000 customers in over 120 countries.

According to Alan Harwood (Senior Manager, Engineering) at Emburse, as a business travel solution, it is critical for them to enable their development and testing teams to operate with complete autonomy and flexibility so that they can deliver features seamlessly. This includes streamlining testing and delivery cycles to increase test automation coverage and outpace the competition in today’s hyper-competitive software industry.

In order to meet the ever-changing market conditions, Emburse use LambdaTest to test faster with confidence as they scale. With LambdaTest, the team was able to eliminate the need to manage the in-house infrastructure and focus on building high-quality test cases.

In-house Selenium Grid infrastructure required continual maintenance

When Alan joined Emburse, his primary responsibility was to design, build, test, deploy, and release web and mobile applications at a global scale.

For testing, they had built a couple of in-house Selenium Grid infrastructure from scratch. They had a monolith approach for deployment with a two-week sprint that used to take between 2 hours and 5 hours. However, they first deployed on the QA, which took up an hour. As a part of their QA checklist, they performed sanity tests, smoke tests, and automated tests using the Selenium framework.

While running the test on a single Selenium Grid used to take up to 45 minutes, to accelerate the test execution time, Alan's team deployed two Selenium Grids so that execution time could be reduced from 45 minutes. As a result, they were able to cut the time to half, which was an outstanding achievement.

However, since each grid was built differently, it was challenging to build and maintain another Selenium Grid from scratch as it was time-consuming and costly. In addition, the grids required frequent maintenance, and a small team was burdened with the maintenance responsibilities.

Test flakiness was the other big area of concern. But, again, this can be attributed to infrastructure-related issues such as network failures, disk errors, and high hardware resource consumption.

Test flakiness is a major concern for organizations that want to scale quickly, and maintaining an in-house test infrastructure is a daunting and super-costly affair. In addition, businesses can't keep up with all the changes when each cluster is built differently.

Emburse quickly realized the need to have a reliable and scalable cloud Selenium Grid to deploy faster to achieve faster TTM (Time to Market).

While looking for a reliable and cost-effective solution, they prepared a checklist, which included a matrix of features and factors, including reliability, test execution time, and cost. Then, after doing a comparative analysis and running tests on different platforms, they came down to two things that mattered to them the most, i.e., Speed and Cost.

LambdaTest provided them with a cost-effective and scalable solution that could scale with their needs while also giving them improved control over their existing infrastructure. Emburse could save significantly by migrating automated testing to LambdaTest cloud infrastructure. With LambdaTest, they can focus more on delivering business value instead of being worried about the infrastructure.

Alan had familiarity with similar solutions, and when searching online for some options to compare, LambdaTest presented a viable option.

Alan Harwood

This is a pretty competitive space. I created a full decision matrix only to find that, assuming equal quality of service, the cost was the differentiating factor that mattered most to us. We did a PoC amongst the top considerations and found LambdaTest to be cheaper and FASTER than the competition.”

- Alan Harwood, Senior Manager, Engineering

LambdaTest helped them get rid of the task of building and maintaining the in-house grid, which eventually helped them cut down the overall test execution time.

Automation testing became indispensable for shorter release cycles as they scaled

While automation testing is part of their Definition of Done in the agile scrum methodology, they were on the lookout for a better-suited automation testing solution that could speed up test execution time and accelerate TTM.

They preferred to be able to use multiple programming languages and frameworks with the Selenium Grid. Since QA would be running E2E automated tests, having a single cloud that can run unit tests would have been an advantage.

Another reason that compelled them to go for a faster test execution was that their business was affected due to the poorly managed infrastructure and legacy platforms.

By using LambdaTest, they were able to perform UI testing seamlessly in the cloud without having the need to maintain an in-house Selenium Grid.

“Now, most of Emburse’s testing happens on LambdaTest, including close to 2100 regression tests and 190 sanity tests across different categories. Regression testing is usually run every day in different time zones to ensure that new feature implementation has not resulted in any breakages in existing functionality.” - Alan Harwood, Senior Manager, Engineering

As a part of the Emburse QA automation practice, the types of testing they do include unit testing, API testing, UI testing, and manual testing. While the frameworks and languages used for UI testing include the following:

  • Selenium framework
  • Selenium Grid
  • Robot framework (Python framework)
  • Python language
  • Jenkins (CI/CD)

While Selenium testing is their primary method for UI testing, Postman is their primary method for API testing. They supplement this with manual testing when automated tests are not applicable.

Their UI regression lifecycle starts with writing tests in the Robot Framework, then committing to GitHub. Jenkins is where they kick off that test run. Next, they use Python to retrieve tests stored in GitHub and convert the Robot Framework keywords into a test that Selenium can understand and use. And then, Selenium WebDriver executes each test using the LambdaTest Selenium Grid.

Selenium Component

According to Alan, migration to LambdaTest took minutes. Then, a new Jenkins job was created with the new endpoints, and it was off to the races. This Jenkins job actually made a copy of the Jenkins job for LambdaTest and pointed to the LambdaTest URL and made it pretty simple for them to get up and running. They didn't have to do too much.

So each sprint included a process from Running full regression suite in staging → Deployment to QA → Running sanity suite in QA → Deployment to production → Running sanity suite in production.

They have created different clusters for different geographies and environments as part of the Sanity suite checklist, which helps them quickly validate the results against each category of tests.

They also leverage LambdaTest to run security scans once the code is executed every night. Since Selenium can also be used to support backend use cases, they used Selenium additionally to run security tests.

“We run our full Selenium regression suite every night in our lower environments. We run our sanity test suite whenever we have a major deployment in our upper environments. Our API tests run on demand with each repository change.” - Alan Harwood, Senior Manager, Engineering

LambdaTest's powerful yet intuitive web-based dashboard allows Alan and his team to run automated tests confidently and faster irrespective of the location, without managing resources internally or requiring any DevOps support.

Reduced test execution time gave the confidence to perform cross browser testing at scale

LambdaTest quickly helped them reduce the test execution time and cost to maintain the test infrastructure. Running automation tests on LambdaTest has enabled them to scale their testing efforts quickly. They saw an average of 20% decrease in the time required to run their regression suite. There was also a visible difference in time spent maintaining the grid. It is now down to zero.

Selenium Component

**2021: Prod Acceptance: 43 mins (down from 54 mins in 2020 owing to the test execution benefits offered by LambdaTest cloud Grid)
**2021: QA Acceptance: 56 mins (includes time spent on troubleshooting)

“The service and product have been consistent and excellent. I’m also looking forward to new developments and was just included in a beta program for some new functionality that I’m excited about.” - Alan Harwood, Senior Manager, Engineering

Currently, they run the entire suite on a single browser but plan to add some variety in the future. Browser coverage is not a challenge for Emburse as of now, as they run the test on the same set of browsers every time. However, they want to explore cross-browser testing opportunities with the LambdaTest platform in 2022 by trying the test on different combinations of browsers and operating systems.

Using LambdaTest cross browser automation on the cloud has allowed Emburse to:

  • Reduce test execution time up to 20% and achieve better code quality.
  • Reduce test execution and infrastructure costs by up to 50%.
  • Focus more on building high-quality test cases by freeing up the internal resources.
  • Run tests from anywhere, at any time, on any device.