At Collabora, we invest a lot of hard work to make LibreOffice's features available in an online environment. Recently we greatly improved the Collabora Online mobile UI, so it's more smooth to use it from a mobile device. While putting more and more work into the software, trying to support more and more different platforms, we need also to spend time improving the test frameworks we use for automatic testing. These test frameworks make sure that while we enrich the software with new features, the software remains stable during the continuous development process.
End-to-end testing in the browser
One step on this road was the integration of cypress.io test framework into Collabora Online code. cypress.io is an end-to-end test run in the browser, so any online application can be tested with it. It mainly allows us to simulate user interaction with the UI and check the event's results via the website's DOM elements. That allows us to simulate user stories and catch real-life issues in the software, so our quality measurement matches the actual user experience.
When I investigated the different alternatives for browser testing I also checked the Selenium test framework. I didn't spend more than two days on that, but I had an impression that Selenium is kind of "old" software, which tries to support many configurations, many language bindings which makes it hard to use and also makes it stuck in the current state, where it is. While cypress.io is a newer test framework, which seems more focused. It is easier to integrate into our build system and easier to use, which is a big plus because it's not enough to integrate a test framework, but developers need to learn how to use it too. I saw one advantage of Selenium: the better browser support. It supports all the main browsers (Chrome, Mozilla Firefox, Safari, Internet Explorer), while cypress.io mainly supports only Chrome, but it improves on this area. Now it has a beta Mozilla Firefox support. So finally I decided to use cypress.io and I'm happy I did that because it nicely works.
So cypress.io is now integrated into the Collabora Online code and we already have 150 tests mainly for mobile UI. As we submit most of our work to upstream, these tests are also available in the community version of the software. It's integrated into the software's GNU make based build system, so a make check will run these tests automatically. This is also part of the continuous integration system, so we can catch any regression instantly, before it actually hits the code. It's recommended to all developers of the online code to get familiar with the test framework, so it will be easier to understand if a test failure indicates an issue in their proposed patch. There are a set of useful notes in the source code, in the readme file: [source_dir]/cypress_test/README. Next to that, I try to add some good advice in the following paragraphs, how to investigate if any cypress test is failing on your patch.