All key user flows are identified as the the most important scenarios that users will be facing. It would be nice to automate all these key flows to surface release blockers as early as possible.
For Android UI test automation, since WP is mostly an Android native app, the first choice would be Espresso, which relies on IdlingResource interface implementation across the app, so Espresso tests could easily register to it for easy synchronized waiting.
However this is not the case for the current WP Android app, manual sleep wait has to be added after the main activity becomes idle. Plus the entire post editor is a WebView, which requires further adoption of Espresso-Web.
We also did a trial for a Appium solution and found WP Android app has well-organized content-desc attribute to most of the key elements, the locators are surprisingly easy to find.
Consider the current selenium based WordPress.com e2e test automation are in Javascript, it would be better to use the same language for the Appium based mobile test automation.
An Appium solution would also make use of the current Sauce Labs infrastructure quickly, while with Espresso, a totally different infrastructure like Firebase needs to be looked into.
As a result, we decided to go with a Javascript Appium solution for the first stage, while we will keep making effort to look into the Android app code base and make a more practical Espresso solution.
Currently, an Appium Javascript solution has been created as in https://github.com/chenchaoyi/wp-android-tests.
It has implemented a complete flow of publishing new post with an image gallery. A screencast of its live action could be found in https://www.dropbox.com/s/fvsc941tufa3fq6/wp_android_appium.mp4.
The structure of the solution is as follows:
├── config │ ├── default.json │ └── testData.json ├── lib │ ├── index.js │ └── pages │ ├── editor.js │ ├── general.js │ ├── mysite.js │ └── signin.js └── test ├── mocha.opts ├── test.js └── wp └── publish.js
Page object convention is followed as all pages are defined under lib/pages, and all test suites are placed under test/wp.