I remember the first time I heard about end-to-end testing (E2E). I was so excited to test an MVC application I was working on. The application had a couple million users logging in each day to use their system. I can remember thinking that E2E testing would solve all of our problems and make our lives easier…
Fast forward a couple of months and I was stuck in a new problem called Selenium. There weren’t many standards, and things quickly got out of control. What seemed like paradise at first was becoming a nightmare. Selecting elements to test was difficult (originally XPATH) and the tests started to take too long to run. Tests were failing intermittently and we spent a lot of our time trying to solve timing issues using custom wait mechanisms. Eventually, we slowly gave up on our hope and dreams that E2E would be the answer.
E2E testing is hard. But it's manageable.
There are many E2E tools out there to choose from. For me, it started with Selenium, then WebDriver, and then Protractor. There are other tools out there, like Nightwatch, and some that I’ve never used or heard of, but they all basically extend from Selenium (and are all faced with the same challenges). The web has changed dramatically over the past 10 years, and the E2E side has struggled to find an effective way to handle it.
The modern web is no longer static pages from server-side rendering, delivered to a client, and subjected to full page refreshes. They quickly evolved to add JavaScript tidbits to help user interactions. This laid the groundwork for libraries like jQuery that allowed us to create more dynamic user experiences, where Ajax calls could bring HTML or data to be injected into our page.
Now we have client-side MVCs, like Angular, React, or Vue.js where almost everything is done on the client using JavaScript.
Why don’t these E2E testing tools work well with the modern web?
First, the present tools are dealing with a snapshot in time and since the data is being delivered dynamically, it’s often not there at the time we start to select elements and make our assertions. This leads to timing issues that cause our tests to fail intermittently and without reason.
Next, we have to have the right data so we can test our edge cases. This often makes our tests much more complicated because we first have to create the data, then locate the data, and test against it. The other option is to have a special database that is seeded with the correct data and is reset when the tests are run each time.
And Finally, Speed is a big problem when running E2E tests, as it can take 4-8 hours to test a large suite of tests. This is far too long to run during the day, as part of the continual integration process, so it gets relegated to an after-hours job. Because it is run after hours, and not a part of the CI build, developers disassociate test failures with their code and chalk it up to timing issues. If we want our E2E tests to be effective, they have to be a part of the CI process.
When tests don’t pass, the error messages are often very cryptic and don’t give good clues for developers to figure out why the test actually failed. To compound the problem, dev tools aren’t available while running the tests, so debugging issues can be almost impossible. This is particularly frustrating to developers — it’s like tying one hand behind their back, and asking them to wrestle a bear while covered in bacon grease.
So if E2E testing is so difficult, why does anyone do it?
The reality is that, because it’s so difficult, it’s almost NEVER done. And, if tests are written, they are rarely run, because the results can’t be trusted. This forces us to write more unit tests to validate our underlying code and hope for the best when it runs in the browser.
This is a recipe for disaster and needs to be fixed.
There has to be a better way to test…
I gave up writing E2E a while back, after getting fed up with wasting my time, but recently I was introduced to a new approach that changed my mind. Cypress.io is a new E2E testing tool, built from the ground up, that makes E2E testing fast, easy, and reliable.
Cypress.io is the missing tool in your toolbox that will help you write tests quickly, debug them, and test your edge cases in the browser. It’s super easy to setup and write your first steps. Install is as easy as:
The first command installs cypress to your local project and the second command runs cypress, which opens the window below.
Cypress.io test window
I’ve created a simple repo that you can use to follow along at:
By default, cypress is going to create a folder named “cypress” in your project and add example tests to your project. This is a great way to learn how cypress works and see examples of how you might write your tests. If you haven’t started your local project, cypress can detect that, and give you a warning message in the cypress window. Once you’ve run ng serve and your project is started, you can kick off the example tests and watch them run.
cypress tests running
Once the tests are finished running, you can select a test, select individual steps, and time-travel through your test to see how each step effects the UI. If an error occurs, you get a verbose error message with hints about why the test failed and what you should look at.
cypress error reporting
Writing E2E tests using Cypress.io
After playing with cypress for a few minutes, I was able to quickly see how it was different from tools I’d used in the past and how I could leverage it to write dependable E2E tests. But what about actually writing tests? I didn’t really want to have to learn a whole new testing syntax. Although it’s different, it’s very similar to the E2E syntax I’d done in Protractor, so the leap was small. The basic premise for a test looks like:
visit->query->action->should
In this example, I visit my site, I find a form that has a class called query-form on it, and I find the first and last input boxes to check their place holders. My test first visits, then queries, and then checks to see what attributes should exist on each.
My first real test
So now that I understand the basics, I’m ready to blow away the examples and create my own tests. I delete the existing examples folder, create a new home-page-spec file, and added a simple test to load my home page.
Cypress automatically detects that I’ve deleted the old example tests and created a new test. When I run it, it opens the browser and loads the page at localhost:4200, where I can see a standard Angular CLI generated homepage.
Sweet success!
Reminders when writing E2E tests
The one thing that most developers forget to do is start their project before running cypress, so they get an error. The first logical thing to do then would be to have cypress run ng serve so my project is always running when I kick off my tests.
This seems very logical, but don’t do it. It’s considered an anti-pattern as cy.exec can only run tasks that eventually exit. The other issue is if the project is already running, then you will have port conflicts.
ProTip: start your project manually.
Notice in that home page test that I don’t specify the url. The best practice is to create a cypress.json file in the root of your project and set your base url. You want to set it to what your local project will default to when serving your pages. This eliminates hardcoded urls and can be overridden when you launch cypress. This will make it easier when testing in different environments.
Here is what the override look like when you want to run the tests in another environment:
npx cypress open --config baseUrl=http://mywaycoolwebsite.com
What else can Cypress.io do?
We have discussed the ability to debug tests, step through tests, and get meaningful error messages - and it could be convincing enough to sway some, but I’m saving the most powerful stuff for last.
The real challenge in E2E testing is dealing with edge cases and setting up seed/magic data so we can test every aspect of our application. This takes enormous amounts of time, makes the tests slow, and creates a fragile world that quickly crumbles.
Prevent Crumbing Code Base by Stubbing Responses
The solution to the problem I just described is to stub our responses from the API. If I have properly written my API, I should already have integration tests that adequately test out the backend and thus, I want to focus on how my application responds when different data is presented.
Cypress allows me to stub responses from my backend so I can always get the data I am expecting and do so super quickly. Since these calls get intercepted, they are blazing fast. So how do I stub a server response? First I start the cypress server and then declare what routes I want to stub as well as their responses.
Now, when my application requests users via a GET call, instead of making the call to the API, I can stub the response to whatever data I want so I can easily test each of my edge cases. This solves the biggest problems that have plagued E2E testing: tests are fast and return consistent results.
Reduce testing bloat with JSON fixtures
It might not jump out at you right away, but writing stubs in the previous example could start to get difficult as the response is hardcoded and becomes bloat within the tests. Fixtures solve this by allowing us to create JSON files with the data we want returned.
We place these in the cypress/fixtures folder and cypress knows where to go looking for them. Imagine the possibilities as you can create multiple user data files to cover every scenario: users.unauthorized.json, users.large.json, users.json, etc…
ProTip: grab real data from the network tab, copy it into a fixture file and then change it as needed.
Reduce testing fragility with wait commands
One of the big issues with E2E in the past has been timing issues. When we wrote E2E tests we would have to make certain assumptions about what should load first and what element we would wait for before querying for other elements. Unfortunately, pages don’t always load the same way or the way that we expect, and this also caused our tests to be fragile.
Cypress introduces a great way to handle this issue as it piggybacks onto the stubbed responses and makes sure that everything is ready. The wait command allows us to pass an array of fixture aliases that it will wait for them to all complete before going to the next step. Since these are stubbed responses, they will return quickly and consistently, which allows us to make sure the page is fully loaded before we begin our assertions.
Remember pulling your hair out wondering why your tests are failing when run in a particular environment, but pass everywhere else. Those days will be quickly over and your test results will be consistent.
E2E testing so convenient it feels like cheating
The number one question/comment I get from people when talking to them about cypress is that it feels like they are cheating. There is this underlying belief that data that isn’t returned from the server isn’t real and that I haven’t really tested my application E2E. I would agree with this if you did 100% stubbed responses and nothing else, but I think we need to change our thinking.
The point of E2E tests is to make sure our UI is going to respond to a user in the way that we expect based on what the user does and what the data is. The purpose of E2E is NOT about testing every possible way of loading data. Once we have loaded the data successfully once, it’s reasonable to assume the application can load it correctly again. We do need different types of data conditions though to test every edge case, so here is my recommendation.
When testing a page using cypress for your E2E tests, write your test with no stubbed responses. This test will serve as a smoke test and validate your integration points. You will want to check for the existence of data, but not much more than that. I usually just check to make sure there are no errors. Once the smoke test is written, the rest of your tests for this page should be stubbed responses where you can dig into every edge case. This is an effective technique for testing that will return consistent results and run very, very quickly.
Cypress.io Testing Summary
Cypress is an amazing tool that solves the E2E testing problems of the past. Our tests become fast, reliable, and complex interactions become manageable. The key to E2E is stubbing out all the different data scenarios so we can focus on making sure our applications respond as expected.
I hope this introduction gets you excited to download and write a few tests with Cypress. There are lots of other tools included to checkout and play with. If we can help you in any way with your testing strategies for E2E or unit testing, please reach out and drop me a line.
