How We Enhanced Our End-to-End Testing with Cypress.io

Automatic testing of web applications is bread and butter for any Quality Assurance specialist. But among so many testing suites available and various opinions about each of them, one can lose his bearings. We've come our way of trying a few end-to-end testing tools and we ultimately chose Cypress. But his article won’t be a paean to Cypress.io (although it might seem like one), neither will I be drawing any comparisons between it and any other testing suites. You can find lots of such content in the Web. 

But over the course of this brief article, I'd like to show you why we even started seeking a new automatic testing suite, how Cypress eventually worked out, and why we find it the best fit to our QA team at Monterail. 

Intrigued whether we've found the Holy Grail of the end-to-end testing tool? Let's dive deeper into our real life case and see yourself. 

The first pick—Nightwatch.js

Before we get to how and why we selected Cypress, we’ll start out by going over some relevant information about the aforementioned case study. The Web app I’m writing about is based on Vue.js (yep, JavaScript again!) + a mock server, and has been in development for over twelve months now. The first automatic testing framework that we decided to use for the project was Nightwatch.js and reached a test coverage level of around thirty percent, covering very basic features, including logging in, navigating between views, and filling out basic forms. 

When the app’s development was advanced enough that we began thinking the next release, our client (working in the network security industry) started asking questions about the stability and security of the application with increasing frequency.

It became clear that a 30% test coverage was not a satisfactory outcome for any involved party.

The problem was that we’ve hit a wall with Nightwatch-based testing. To give an example, the Selenium Web Driver was very issue-prone, we’ve had problems with test stability, and we were rather dissatisfied with the pace of writing new tests. It quickly became apparent to us that if we were to boost our test coverage to include most of the features, we would have to find an alternative testing suite. And a good one at that.

The second try—Cypress.io

I could write here at length about the pros and cons of Cypress and explain, in minute detail, the process that led to its selection, but the truth is that we needed a JavaScript-based alternative to Selenium that would help us deal with our current problems and quickly boost our test coverage numbers.

After reviewing relevant posts across a number of blogs and forums, and poring over roadmaps for a couple of different testing suites, we ultimately decided to give Cypress a shot. Initially, we had our app tested by two separate tools. Only after we were certain that Cypress would become our main suite did we rewrite the remaining tests for the chosen framework.

After spending a couple of months with Cypress, I have a handful of pros and cons that I’d like to share. The list of advantages is long so brace yourself.

Why use Cypress.io?

It’s not based on Selenium.

Despite the fact that the technology is still very popular, and the community and documentation have matured, there is no denying that testing modern Web apps requires software capable of keeping up with all the recent trends in tech. Selenium can handle a spectrum of platforms, browsers, and languages (a huge advantage of that particular technology), but if you take a look at the image below and appreciate that automatic testing really can be that simple, you’ll realize that Cypress at the very least deserves a chance. 

 How Cypress works, Source: Cypress.io

Easy and effortless installation and updating.

If you’re using Node.js, installing Cypress in your project will require just a single line in the console:

 npm install cypress

and presto! :) Obviously, proper configuration of the suite and the tests themselves to fit your your project is not that simple. But the truth is, that Cypress will allow you to run some basic tests in minutes (seconds even?) after completing the installation. New version available? Type innpm install --save-dev cypress@3.1.0 and you’re ready to go.

Less problems with dependencies.

At least we’ve never had Cypress produce any issues with nodes, their versions, and related dependencies. Cypress simply works.

It’s based on JavaScript, the language of the Web.

If we are to believe the team behind it, Cypress was created by developers for developers. And all the devs on our staff (have to) know JavaScript. Bring these two facts together, and you’ll be getting a testing suite that’s approachable not only to professional QAs, but also to junior developers who are only beginning their adventure with E2E tests. The Cypress API lacks a feature you deem necessary? That’s usually not a problem, as you can easily write it yourself in pure JS and then implement in your tests down the line.

It has an awesome dashboard and preview window.

Well, it looks exactly as it sounds. Just look at the screenshot below. It gives you a precise overview of what’s being tested, how long will it take, how many tests have already been completed, and how many failed. Pausing, restarting, timers, previews, logs—everything’s there, in a neat and digestible package sitting in your browser window.

 Running tests in Cypress.io, Source: Cypress.io

Broad range of debug and test tracking options.

The list includes integration with dev tools, screenshots, snaps etc. The clear and informative dashboard is cool, but if you add in the ability to monitor every step and assertion of your tests (including snaps and application states in any given moment of the test), as well as additional information available to you through the dev tools you’ll quickly come to realize that Cypress has really elevated the debugging of your tests (and your application, too) to a higher level.

 Cypress.io dashboard

Headless option and ease of integrating with CI.

At present, Cypress can be integrated with around 15 CI providers. At Monterail, Google Cloud Platform (with Docker) is the CI tool of choice for this project. Thanks to the headless feature, we were able to integrate our testing with the Google Cloud Platform. At this point, tests take around 10 minutes. Recently, Cypress also introduced the ability to parallelize test runs. It should help us cut down the time it usually takes to run a test to around 5 minutes, at least that’s the results that the Cypress staff have been touting. Additionally, we were able to launch the feature that saved an app screenshot whenever the test managed to produce an error. A very useful feature, as sometimes logs are not enough to deduce what went wrong, so it’s nice to have another way of looking at what was happening with the app in a given moment.

Network traffic control.

By this I mean detection, debugging, verification, waiting for requests. Naturally, these features are nothing new in the automatic testing world. But working with network requests in Cypress is more than friendly. The Cypress dashboard, along with the dev tools console, allows us to check out each request at any moment in the test. Quickly and cleanly. Cypress also allows us to halt the tests until a given request is fulfilled by the server. 

Network traffic control in Cypress.io, Source: Docs.Cypress.io

DevTools Console output of XHR Request sent by Cypress Runner

Eliminating all sorts of timers.

Cypress features a built-in mechanism that handles waiting for DOM elements (all we have to do is set the maximum waiting time, which is set at 10,000ms by default). This brings two advantages:

• all features like sleep or wait are more or less eliminated from the tests. The result?
• tests are much more stable, the tests are running much quicker. Each subsequent step of the test is launched immediately after they appear in the DOM.

Good documentation, growing community, active support forums.

Although Cypress is still a fairly new endeavor, we had no trouble finding help online throughout the nearly twelve months we already spent writing E2E tests in Cypress. The official API and the documentation look really good. The knowledge base and user activity on the official support forum and relevant corners of Stack Overflow are also very promising.

Real-time reloads.

Saving the test file automatically launches the test in the dashboard. A small thing, but highly enjoyable, as it allows you to concentrate on writing code rather than running tests.

Easy, intuitive syntax (Mocha and Chai are all there).

No one with any experience in writing automatic tests and basic knowledge of JS should have any trouble with handling Cypress syntax (I know I haven’t had any). 

A plethora of selectors.

Cypress gives us the ability to work with “jQuery-like” selectors (although it’s not the only option). If we were to add to that Cypress’ built-in commands, such as .parent(), .parentUntil(),next(),.children(),.closest(),.contains(content),.eq(index), .find(selector),first(), we’d quickly realize that we’re capable of capturing 99% of the elements in our app swiftly (and clearly), without having to add new classes or attributes.

The downsides of Cypress

Alright, it’s never all roses. Cypress tends to lack in these areas:

No cross-browser testing capabilities.

If you need to test apps in Firefox, you won’t be able to run these tests with Cypress (at least not as of this writing). It’s a considerable drawback, but we didn’t need cross-browser testing capabilities so much that it would force us to stay with our previous testing suite. At this point, Cypress only supports Chrome and Electron.

No native event handling capabilities.

Such as upload, download file, tab key, etc. This is one feature that we genuinely miss, and its lack gets more noticeable as the app grows. Native events don’t actually handle our killer features, but we’d feel more secure if we could at least test them in our end-to-end tests.

The Holy Grail of the end-to-end testing?

As of this writing, both cons are being actively handled by the open-source community, as per the roadmap. It seems that soon enough we’ll be able to both tests our apps in Firefox and Edge, as well as use the browsers’ native events in testing.

One of the reasons Cypress remains so difficult to ignore is the fact that it’s a project with a very strong open-source community backing.

Reading the forums, the project documentation, and seeing the roadmap, you can’t help but get the impression that it’s being developed by people perfectly aware of what a fast, intuitive, and easy Web app testing engine should look like. Before committing time and energy to learning an emerging technology, it’s good practice to establish whether the project itself is moving in the right direction. 

Cypress is still a relatively new piece of technology. At this point, it’s hard to compare it against the capabilities offered by more mature testing suites. However, in case of our project, it worked splendidly. Writing new tests is no longer a challenge, and we’re able to extend the test coverage to cover new and upcoming features basically in real time. And it is our expectation that soon enough we will be able to put the suite to the test in other projects, too, to ascertain its effectiveness in other cases.