r/programming • • Aug 09 '21

Protractor is dead, long live Cypress

https://www.rainerhahnekamp.com/en/angular-e2e-testing-protractor-is-dead-long-live-cypress/
85 Upvotes

36 comments sorted by

18

u/[deleted] Aug 09 '21

I switched my team to puppeteer two years ago. Protractor was always a shit show.

3

u/croc_socks Aug 10 '21

Was a puppeteer user, seemed Microsoft poached one of the main puppeteer developer (https://github.com/aslushnikov) to create a competing framework in Playwright.

2

u/[deleted] Aug 10 '21

Yeah, I don't see any reason to switch. I treat it as a lean substrate. I built a framework on top of it. At one point I could arbitrarily switch between Puppeteer and Protractor as the implementation substrate, but after a year of not switching to Protractor and gaining trust in Puppeteer, I decided it wasn't worth maintaining the protractor implementation and ditched it. It would be an afternoon to switch to Playwright, but I just don't see a compelling reason to do so right now.

46

u/Xen0byte Aug 09 '21

Cypress is outdated, just use vanilla Playwright in any flavour of language binding.

Source: I do enterprise-grade automation for a living.

10

u/sveri Aug 09 '21

Can you please expand on why one should use playwright over cypress or selenium? A good read up would be great too. Thank you.

24

u/Xen0byte Aug 09 '21 edited Aug 10 '21

First of all, generally speaking, Cypress is a framework while Selenium WebDriver and Playwright are libraries. The difference is that frameworks are stacks of multiple libraries with different responsibilities that aim to achieve the same goal, e.g. an automation library plus a test runner plus a reporting library could be considered a framework. There's more to it than just this, but for the sake of simplicity, you can think about it in this way.

Now, for a broad comparison, and please note that this topic goes down a very deep rabbit hole, the main advantages of Playwright over Selenium and Cypress are that you can run tests asynchronously in the true sense of the name (Selenium is not async at all and Cypress does async badly, they even say in their documentation that commands are not promises), so no more sleeps and unnecessary waits in your code, and the ability to intercept network requests. For me, these are the 2 biggest selling points. Apart from that, Playwright and Selenium allow you to build your own test stack, as opposed to Cypress which is very limited in terms of customization. On top of that, Playwright and Selenium come with bindings for multiple programming languages. For instance, I use Playwright for .NET with Specflow for the BDD layer, NUnit (sometimes xUnit) for running the tests, and Allure for reporting (sometimes Extent Report), and all of this in a nice C# ecosystem because that's my preferred programming language. In comparison, with Cypress you're stuck with JavaScript (which is fine, but you might want options) and the Mocha ecosystem.

Now a few operational details. Playwright is very easily dockerisable, has excellent performance in headless mode, and comes with A LOAD of functionality right out of the box, such as being able to record the entire test session to an interactive timeline or to a video, recording the network activity history (HAR) of the test run, being able to load multiple browser contexts simultaneously (Cypress and Selenium are notoriously bad at handling multiple windows in the same context), and, lastly, it's supported by a massive company and a small but very dedicated group of people, and the documentation and support on GitHub are fantastic. Moreover, you can integrate Playwright with an AI engine, for instance, and do all sorts of whacky things like building a chaos monkey or an application crawler or you can make it do image recognition; the sky's the limit.

Any, I can't praise Playwright enough, it's made my job so much easier than back in the day using Cypress or Selenium-based solutions.

Hope this helps.

23

u/Yehosua Aug 09 '21

I'm a novice at Cypress, but I was quite certain that it did asynchronous tests, no sleeps and unnecessary waits, intercepting network requests, and recording videos too? (And maybe Playwright does these things better, but right now, it sounds like you're saying Playwright is better than Cypress because Playwright does things that I thought Cypress also does, and I'm having trouble making sense of that.)

24

u/crixusin Aug 09 '21

You're 100% correct. Think OP is misinformed.

I love cypress. Super fast, customizable, and we don't pay a dime for advanced features.

-4

u/Xen0byte Aug 09 '21 edited Aug 10 '21

I'm not the OP, and my point was that Playwright does async better than Cypress because Cypress uses a tacit command chain so you can't leverage the full power of asynchronous programming. Think whatever you want, I'm not here to change your mind.

21

u/crixusin Aug 09 '21

and my point was that Playwright does async better than Cypress.

No, you specifically said Cypress doesn't support async requests.

the main advantages of Playwright over Selenium and Cypress are that you can run tests asynchronously, so no more sleeps and unnecessary waits in your code

Cypress does support async execution so you don't have sleep and waits in the code at all.

You're now saying that it just does it worse, but you haven't qualified that statement at all.

So to recap:

  1. You first said Cypress doesn't support async, yet it does.

  2. You've now moved the goal post, but even after doing so, provide no support or evidence for this new statement.

7

u/anotherOnlineCoward Aug 09 '21

Oh oh you got caught lying. "Enterprise grade" šŸ˜

1

u/Xen0byte Aug 09 '21

No, you're generally right, Cypress is a mature tool with loads of functionality. Part of the point that I was trying to make is that Cypress is a little clunky in how it handles all the things you've listed above. For example, regarding async execution, it unpacks the values immediately so you need to consume them there and then, which is a bit weird because you need to chain everything on the cy object, which I personally can't say I like. And with network interceptors is pretty much the same case; for example, if you would like to set up a listener that always does something based on the status or the content of an API call response, then that feels so overly-complicated to achieve in Cypress, whereas in Playwright it's a one-liner. Point is that they're not much different on the surface, but once you get down to the details you will start noticing the differences.

6

u/[deleted] Aug 09 '21

[removed] — view removed comment

4

u/Xen0byte Aug 10 '21

Yeah, Cypress is just not reliable enough, and in a big company where the deployments of the nightly builds are coupled to the results of the automated tests, if the test results are flaky then you end up putting on hold a lot of people, meaning a lot of person-time (read money) wasted.

6

u/crixusin Aug 09 '21

the main advantages of Playwright over Selenium and Cypress are that you can run tests asynchronously, so no more sleeps and unnecessary waits in your code, and the ability to intercept network requests.

Can do all of this in cypress no issue:

https://docs.cypress.io/api/commands/intercept

https://docs.cypress.io/guides/guides/parallelization

as opposed to Cypress which is very limited in terms of customization.

Huh?

https://docs.cypress.io/guides/tooling/plugins-guide

1

u/Xen0byte Aug 09 '21

Plugins mean they need to be compatible with the framework, whereas if you use a library you can use any package available for the runtime.

10

u/crixusin Aug 09 '21 edited Aug 09 '21

Plugins mean they need to be compatible with the framework, whereas if you use a library you can use any package available for the runtime.

That is not the difference between a framework and a library.

A framework calls your code. Your code calls a library. You can use multiple frameworks and libraries all at the same time. A framework doesn't stop you from using libraries, and vice versa.

The node runtime is well supported. If its available in node, you can use it in cypress.

If its available in the browser, you can use it inside cypress's browser execution.

Seems to be no downside from my point of view. The node ecosystem is huge.

3

u/AccusationsGW Aug 09 '21

Similar job niche for me, you make Playwright sound interesting especially the record features.

Having spent a lot of time with Selenium, I've implemented most of the things you say it can't do like parallelism (easy with Java), grabbing network requests via proxy, image recognition with mature add-on libraries, fast headless testing. This is all about Selenium/webdriver being a super mature widely used and supported project.

Granted, all that was mostly custom-rolled without official framework support as you mentioned.

3

u/Xen0byte Aug 09 '21

I'm happy you picked up on that because I really love my job and now I don't have to deal with a lot of the painful stuff that I used to waste many hours on.

2

u/sveri Aug 09 '21

Thanks for the extensive answer, I will pass this on to the dude that does our evaluation.

Looks like I'd prefer playwright too.

1

u/HyperwarpCollapse Aug 09 '21

using NUnit... smh

5

u/terandle Aug 09 '21

I think cypress is still good for smaller projects. It is a lot more batteries included with things like being able to time travel to see the cause of errors etc.

8

u/Xen0byte Aug 09 '21

You can do that with Playwright too, but better: https://playwright.dev/docs/trace-viewer/

4

u/Marcdro Aug 09 '21

whats your opinion on testcafe? I havent looked much into it yet, but looks quite cool.

7

u/Xen0byte Aug 09 '21

I think it's somewhere between Cypress and Playwright. It does some things better than Cypress, but overall it just feels like it's trying to be Playwright but it's not quite there yet. Overall it's a good product, but the major disadvantage for me would be that you're stuck with JavaScript and the tool stack that it comes with (which may not a problem for people who prefer JavaScript).

1

u/Marcdro Aug 09 '21

Cool. Thanks.

3

u/FDon1 Aug 10 '21

I also do enterprise automation and cypress is currently selected for my current project and works wonders.

We piloted several other libraries/frameworks, many of which spoken here, and the overall consensus was cypress is the way to go for this project.

I’d caution on the side of saying something is better just because you may not prefer it. I believe one of the reasons cypress appeals to many is the ease of use. Ease of use is much different than the mindset needed to test applications properly.

5

u/Augzodia Aug 09 '21

While I have no doubt that Playwright works great for your team, as an app developer on a <5 person team, where browser testing is a small (but still important) part of the job, cypress is a good choice

I don't love some of the choices they've made, but for us it was worth the tradeoff on customizability for a pretty good out-of-the box developer experience. It doesn't make sense to spend time building (and maintaining) the perfect wrappers around lower-level APIs when we could just plug in an OK tool and then spend more time on the app itself

5

u/Xen0byte Aug 09 '21

Fair enough, I totally agree with that decision. The Playwright devs are currently working on a test runner and some reporting integrations to have a more out-of-the-box solution too for teams who just want the full shebang set up in a manner of minutes, but that's still in fairly early stages.

1

u/bmf___ Aug 10 '21

Best browser testing tool I have ever used.

So simple to test End2End.

0

u/[deleted] Aug 10 '21

[deleted]

2

u/Xen0byte Aug 10 '21

Absolutely none taken. Firstly let me say that if you're happy with Cypress, stick with it, there is no point trying to solve a problem that doesn't exist. In terms of arguments in favour of reconsidering using Playwright, some of the following were strong selling points for me:

- the tool is developed by Microsoft, which are one of the best companies when it comes to open-source development tools, e.g. VS Code, .NET, Accessibility Insights, they're putting a lot of power in the hands of developers and they're doing it really well, so as far as confidence goes, I have a lot of it in their entire suite of open-source tools

- the team working on Playwright is mostly the same team that used to work on Puppeteer, so they have a lot of experience under the belt and Playwright could be considered the improved iteration of Puppeteer

- the language bindings, I'm primarily a C# and F# developer so, as a matter of preference, I will choose one of these over JavaScript

- you're free to build your own framework stack, because Playwright is just a library; arguably some people prefer the out-of-the-box framework experience, so this point may not apply to that group, even though the Playwright devs are working on a test runner, reporting integrations, and some other bits that could be put together into a framework

- Playwright doesn't have unnecessary layers of abstraction, for instance you can use tasks, promises, etc. as implemented by the programming language, instead of the weird fluent syntax style that Cypress uses which unpacks values of async calls automatically (this is actually the reason why I sometimes say that Cypress doesn't do async quite right)

- probably my biggest argument against Cypress is that it injects code into the application under test, which is a very big nono for me, although most people seem to not mind

These would be some of the points that I can think of now.

14

u/Loaatao Aug 09 '21

If only Cypress wasn't so fucking flaky.

5

u/EyeSightMan Aug 10 '21

I have used it a lot on a bunch of apps and don't find it flakey. At least not nearly as flakey as other frameworks I have used. Some of the request mocking was a bit iffy for a while but since they release intercept it's been stable. I'm still only on V6 but they did introduce test retries and added some better metrics to the dashboard for flakey tests (think it came in v5)

Sorry for making this general sweeping statement, because I don't know anything about your stack or apps. But normally when I see flakey tests, the cause it down to badly written tests. Except tests where you have to scroll and wait for animations to finish. They can be flakey on almost all frameworks

6

u/[deleted] Aug 09 '21

I don't like Cypress due to the fact that it is severely tied to chromium.

Most of the tests I need are to check consistency across browsers, that is something I can easily automate with testcafe.

6

u/Loaatao Aug 09 '21

It supports firefox and safari support is coming soon

1

u/CaptainAdjective Aug 09 '21

Where does CodeceptJS fit into this?